CeLLaTe-tapt_ulmfit_dropout_whole_word-LR_1e-06

This model is a fine-tuned version of microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.2115
• Accuracy: 0.7384

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-06
• train_batch_size: 16
• eval_batch_size: 16
• seed: 3407
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-06 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.06
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
1.417	1.0	21	1.3054	0.7242
1.4427	2.0	42	1.2771	0.7220
1.4372	3.0	63	1.2604	0.7231
1.4313	4.0	84	1.2655	0.7309
1.4279	5.0	105	1.2783	0.7278
1.4423	6.0	126	1.2915	0.7240
1.3965	7.0	147	1.2815	0.7287
1.4224	8.0	168	1.2791	0.7277
1.4191	9.0	189	1.2835	0.7270
1.4469	10.0	210	1.2283	0.7312
1.4211	11.0	231	1.2734	0.7269
1.4104	12.0	252	1.2595	0.7279
1.4034	13.0	273	1.3035	0.7225
1.4017	14.0	294	1.2485	0.7315
1.424	15.0	315	1.2264	0.7400
1.43	16.0	336	1.2705	0.7272
1.4092	17.0	357	1.2663	0.7302
1.3956	18.0	378	1.2395	0.7315
1.3756	19.0	399	1.2697	0.7264
1.4096	20.0	420	1.2773	0.7276
1.3797	21.0	441	1.2720	0.7324
1.4215	22.0	462	1.2762	0.7312
1.3908	23.0	483	1.2269	0.7320
1.3587	24.0	504	1.2476	0.7333
1.3807	25.0	525	1.2124	0.7350
1.3917	26.0	546	1.2850	0.7261
1.4013	27.0	567	1.2109	0.7440
1.4236	28.0	588	1.2679	0.7312
1.3988	29.0	609	1.2919	0.7282
1.3988	30.0	630	1.2626	0.7327
1.3863	31.0	651	1.3055	0.7246
1.4063	32.0	672	1.2723	0.7307
1.3824	33.0	693	1.2363	0.7335
1.4027	34.0	714	1.2655	0.7313
1.3938	35.0	735	1.2437	0.7329
1.378	36.0	756	1.2797	0.7266
1.3678	37.0	777	1.2608	0.7280
1.389	38.0	798	1.2706	0.7294
1.4134	39.0	819	1.2581	0.7290
1.3866	40.0	840	1.2719	0.7294
1.378	41.0	861	1.2304	0.7350
1.3844	42.0	882	1.2708	0.7277
1.3749	43.0	903	1.2824	0.7266
1.4224	44.0	924	1.2357	0.7339
1.3705	45.0	945	1.2464	0.7342
1.3521	46.0	966	1.2224	0.7324
1.3956	47.0	987	1.2440	0.7358
1.3941	48.0	1008	1.2374	0.7359
1.3796	49.0	1029	1.2603	0.7293
1.3945	50.0	1050	1.2365	0.7353
1.3874	51.0	1071	1.2675	0.7275
1.385	52.0	1092	1.2642	0.7328
1.3787	53.0	1113	1.3000	0.7241
1.3585	54.0	1134	1.2610	0.7331
1.379	55.0	1155	1.2489	0.7302
1.3917	56.0	1176	1.2527	0.7306
1.3847	57.0	1197	1.2648	0.7325
1.4298	58.0	1218	1.2781	0.7289
1.3857	59.0	1239	1.2128	0.7406
1.3954	60.0	1260	1.2630	0.7280
1.3726	61.0	1281	1.2678	0.7317
1.3955	62.0	1302	1.2821	0.7316
1.3881	63.0	1323	1.2372	0.7377
1.3775	64.0	1344	1.2855	0.7280
1.3985	65.0	1365	1.2628	0.7302
1.3641	66.0	1386	1.2816	0.7269
1.4088	67.0	1407	1.2770	0.7269
1.3488	68.0	1428	1.2758	0.7223
1.34	69.0	1449	1.2518	0.7284
1.3589	70.0	1470	1.2115	0.7341
1.3732	71.0	1491	1.2959	0.7261
1.3846	72.0	1512	1.2925	0.7275
1.3888	73.0	1533	1.2442	0.7337
1.368	74.0	1554	1.2889	0.7292
1.3676	75.0	1575	1.1854	0.7444
1.3577	76.0	1596	1.2998	0.7287
1.3621	77.0	1617	1.2169	0.7372
1.387	78.0	1638	1.2295	0.7333
1.3618	79.0	1659	1.2466	0.7363
1.3864	80.0	1680	1.2806	0.7290
1.371	81.0	1701	1.2339	0.7342
1.3785	82.0	1722	1.2343	0.7330
1.3636	83.0	1743	1.2524	0.7312
1.3936	84.0	1764	1.2523	0.7316
1.3771	85.0	1785	1.2627	0.7277
1.3557	86.0	1806	1.2166	0.7362
1.3803	87.0	1827	1.2356	0.7362
1.3954	88.0	1848	1.2200	0.7394
1.3642	89.0	1869	1.2566	0.7294
1.3681	90.0	1890	1.2196	0.7394
1.3699	91.0	1911	1.2475	0.7311
1.3568	92.0	1932	1.1959	0.7394
1.3609	93.0	1953	1.2606	0.7330
1.3858	94.0	1974	1.2737	0.7283
1.3874	95.0	1995	1.2670	0.7307
1.3781	96.0	2016	1.2187	0.7357
1.3826	97.0	2037	1.2555	0.7310
1.3553	98.0	2058	1.2408	0.7334
1.3586	99.0	2079	1.2193	0.7379
1.3705	100.0	2100	1.2115	0.7384

Framework versions

• Transformers 4.48.2
• Pytorch 2.4.1+cu121
• Datasets 3.0.2
• Tokenizers 0.21.0