CeLLaTe-tapt_ulmfit_dropout_whole_word-LR_2e-05

This model is a fine-tuned version of microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.2217
• Accuracy: 0.7348

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 3407
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-06 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.06
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
1.4168	1.0	21	1.3051	0.7242
1.4403	2.0	42	1.2747	0.7225
1.4307	3.0	63	1.2563	0.7256
1.4205	4.0	84	1.2579	0.7326
1.4102	5.0	105	1.2684	0.7301
1.4171	6.0	126	1.2760	0.7276
1.3624	7.0	147	1.2583	0.7317
1.3795	8.0	168	1.2537	0.7326
1.3705	9.0	189	1.2584	0.7315
1.3941	10.0	210	1.2015	0.7376
1.3592	11.0	231	1.2455	0.7333
1.3463	12.0	252	1.2337	0.7329
1.3398	13.0	273	1.2741	0.7272
1.3268	14.0	294	1.2216	0.7365
1.3475	15.0	315	1.1982	0.7431
1.3475	16.0	336	1.2407	0.7319
1.3237	17.0	357	1.2356	0.7354
1.3032	18.0	378	1.2140	0.7361
1.2792	19.0	399	1.2441	0.7305
1.307	20.0	420	1.2496	0.7336
1.2778	21.0	441	1.2452	0.7348
1.3144	22.0	462	1.2459	0.7332
1.2788	23.0	483	1.2010	0.7389
1.2429	24.0	504	1.2262	0.7367
1.271	25.0	525	1.1833	0.7366
1.2758	26.0	546	1.2595	0.7298
1.2772	27.0	567	1.1970	0.7437
1.2958	28.0	588	1.2456	0.7316
1.2674	29.0	609	1.2686	0.7301
1.2628	30.0	630	1.2471	0.7334
1.25	31.0	651	1.2833	0.7276
1.2623	32.0	672	1.2441	0.7330
1.2365	33.0	693	1.2179	0.7359
1.2513	34.0	714	1.2510	0.7320
1.2423	35.0	735	1.2279	0.7320
1.2286	36.0	756	1.2647	0.7302
1.2167	37.0	777	1.2505	0.7347
1.2287	38.0	798	1.2558	0.7304
1.2511	39.0	819	1.2422	0.7336
1.2246	40.0	840	1.2565	0.7293
1.2145	41.0	861	1.2175	0.7364
1.216	42.0	882	1.2547	0.7307
1.2098	43.0	903	1.2632	0.7300
1.2505	44.0	924	1.2158	0.7348
1.2034	45.0	945	1.2304	0.7341
1.1799	46.0	966	1.2105	0.7348
1.2212	47.0	987	1.2299	0.7384
1.2175	48.0	1008	1.2231	0.7384
1.2017	49.0	1029	1.2540	0.7335
1.2096	50.0	1050	1.2345	0.7359
1.2066	51.0	1071	1.2665	0.7298
1.2044	52.0	1092	1.2558	0.7337
1.194	53.0	1113	1.2909	0.7272
1.1784	54.0	1134	1.2468	0.7338
1.1953	55.0	1155	1.2440	0.7321
1.2025	56.0	1176	1.2436	0.7300
1.197	57.0	1197	1.2521	0.7339
1.2293	58.0	1218	1.2644	0.7275
1.1934	59.0	1239	1.2097	0.7423
1.2046	60.0	1260	1.2561	0.7329
1.1764	61.0	1281	1.2666	0.7310
1.2009	62.0	1302	1.2778	0.7300
1.1888	63.0	1323	1.2321	0.7368
1.1818	64.0	1344	1.2877	0.7275
1.1988	65.0	1365	1.2585	0.7304
1.1682	66.0	1386	1.2776	0.7301
1.2029	67.0	1407	1.2725	0.7281
1.1549	68.0	1428	1.2764	0.7249
1.142	69.0	1449	1.2492	0.7312
1.1588	70.0	1470	1.2126	0.7345
1.1725	71.0	1491	1.3000	0.7223
1.1821	72.0	1512	1.2860	0.7281
1.1874	73.0	1533	1.2443	0.7321
1.1627	74.0	1554	1.2805	0.7293
1.1601	75.0	1575	1.1849	0.7454
1.1522	76.0	1596	1.3093	0.7252
1.1592	77.0	1617	1.2005	0.7401
1.1791	78.0	1638	1.2351	0.7326
1.1555	79.0	1659	1.2516	0.7333
1.1775	80.0	1680	1.2923	0.7275
1.1594	81.0	1701	1.2398	0.7318
1.1693	82.0	1722	1.2337	0.7342
1.1521	83.0	1743	1.2560	0.7330
1.1789	84.0	1764	1.2464	0.7318
1.1614	85.0	1785	1.2607	0.7256
1.1455	86.0	1806	1.2155	0.7377
1.1598	87.0	1827	1.2383	0.7348
1.1782	88.0	1848	1.2254	0.7371
1.1572	89.0	1869	1.2595	0.7315
1.1586	90.0	1890	1.2221	0.7357
1.1581	91.0	1911	1.2532	0.7323
1.1429	92.0	1932	1.2009	0.7354
1.151	93.0	1953	1.2633	0.7355
1.1729	94.0	1974	1.2735	0.7290
1.1725	95.0	1995	1.2702	0.7290
1.1608	96.0	2016	1.2167	0.7352
1.1713	97.0	2037	1.2516	0.7337
1.1418	98.0	2058	1.2411	0.7327
1.1519	99.0	2079	1.2157	0.7403
1.1559	100.0	2100	1.2217	0.7348

Framework versions

• Transformers 4.48.2
• Pytorch 2.4.1+cu121
• Datasets 3.0.2
• Tokenizers 0.21.0