Lora training on FLUX.1 dev

Question

I have one huge problem of training Lora on Flux..

So, I was using kohya, simpleTuner and want to train a LoRa with my images (dataset of my photos)

and my problem is that any character (woman, boy, man) starts to look like my data-set images. Not sure what am I doing wrong here.. I'm using trigger word and man.

I heard that it is useless to use regularisation folder for LoRa..

Any advices how to train LoRa for such cases?

I used these configs for simple-tuner

main configs.

{
    "--resume_from_checkpoint": "latest",
    "--data_backend_config": "config/multidatabackend.json",
    "--aspect_bucket_rounding": 2,
    "--seed": 42,
    "--minimum_image_size": 0,
    "--disable_benchmark": false,
    "--output_dir": "output/models-fifth",
    "--lora_type": "standard",
    "--lora_rank": 16,
    "--max_train_steps": 3500,
    "--num_train_epochs": 0,
    "--checkpointing_steps": 400,
    "--checkpoints_total_limit": 20,
    "--model_type": "lora",
    "--pretrained_model_name_or_path": "black-forest-labs/FLUX.1-dev",
    "--user_prompt_library": "config/user_prompt_library.json",
    "--model_family": "flux",
    "--train_batch_size": 1,
    "--gradient_checkpointing": "true",
    "--caption_dropout_probability": 0.0,
    "--resolution_type": "pixel_area",
    "--resolution": 1024,
    "--validation_seed": 42,
    "--validation_steps": 400,
    "--validation_resolution": "1024x1024",
    "--validation_guidance": "3.5",
    "--validation_guidance_rescale": "0.0",
    "--validation_num_inference_steps": "20",
    "--validation_prompt": "a photo of zikki. man",
    "--mixed_precision": "bf16",
    "--optimizer": "adamw_bf16",
    "--learning_rate": "1e-4",
    "--lr_scheduler": "constant",
    "--lr_warmup_steps": 100,
    "--base_model_precision": "int8-quanto",
    "--validation_torch_compile": "false"
}

and dataset configs

   {
        "id": "my-dataset-512",
        "type": "local",
        "instance_data_dir": "/home/vic/Desktop/work/simple_tuner_standard/datasets/my-dataset",
        "crop": false,
        "crop_style": "center",
        "minimum_image_size": 128,
        "resolution": 512,
        "resolution_type": "pixel_area",
        "repeats": 2,
        "metadata_backend": "discovery",
         "caption_strategy": "textfile",
        "cache_dir_vae": "cache//vae-512",
        "instance_prompt": "zikki. man"
    },