[2026-10-16 13:57:50] Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
[2026-10-16 13:57:50] '--disable-cuda-graph' is deprecated and will be removed in a future release. Use '--cuda-graph-backend-{decode,prefill}=disabled' instead.
[2026-10-16 13:57:50] Registered linear attn model: config=GigaChat35Config, backend=GDNAttnBackend, archs=['GigaChat35ForCausalLM', 'GigaChat35ForCausalLMNextN']
/private/tmp/sglang-research/sglang-v0.5.22/python/sglang/srt/layers/quantization/awq/awq.py:53: UserWarning: Only CUDA, HIP and XPU support AWQ currently.
  warnings.warn(f"Only CUDA, HIP and XPU support AWQ currently.")
/private/tmp/sglang-research/sglang-v0.5.22/python/sglang/srt/layers/quantization/gguf.py:69: UserWarning: Only CUDA, MUSA and NPU support GGUF quantization currently.
  warnings.warn(f"Only CUDA, MUSA and NPU support GGUF quantization currently.")
[2026-10-16 13:57:52] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit "HTTP/1.1 200 OK"
[2026-10-16 13:57:52] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit "HTTP/1.1 200 OK"
[2026-10-16 13:57:52] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit "HTTP/1.1 200 OK"
[2026-10-16 13:57:53] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit "HTTP/1.1 200 OK"
[2026-10-16 13:57:53] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit "HTTP/1.1 200 OK"
[2026-10-16 13:57:53] mlx_q4 quantization is not fully optimized yet. The speed can be slower than non-quantized models.
[2026-10-16 13:57:53] Attention backend not specified. Use torch_native backend by default.
[2026-10-16 13:57:53] Cuda graph is disabled because of using torch native attention backend
[2026-10-16 13:57:53] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit "HTTP/1.1 200 OK"
/private/tmp/sglang-research/sglang-v0.5.22/python/sglang/kernels/ops/attention/fla/utils.py:223: UserWarning: Triton is not supported on current platform, roll back to CPU.
  warnings.warn(
[2026-10-16 13:57:54] Fail to set RLIMIT_STACK: current limit exceeds maximum limit
[2026-10-16 13:57:54] server_args={'model_path': 'mlx-community/Qwen3-0.6B-4bit', 'tokenizer_path': 'mlx-community/Qwen3-0.6B-4bit', 'tokenizer_mode': 'auto', 'tokenizer_backend': 'huggingface', 'tokenizer_worker_num': 1, 'detokenizer_worker_num': 1, 'skip_tokenizer_init': False, 'load_format': 'auto', 'model_loader_extra_config': '{}', 'trust_remote_code': False, 'context_length': None, 'is_embedding': False, 'enable_multimodal': None, 'revision': None, 'model_impl': 'auto', 'model_config_parser': 'auto', 'json_model_override_args': '{}', 'dtype': 'auto', 'quantization': None, 'quantization_param_path': None, 'kv_cache_dtype': 'auto', 'enable_fp32_lm_head': False, 'modelopt_quant': None, 'modelopt_checkpoint_restore_path': None, 'modelopt_checkpoint_save_path': None, 'modelopt_export_path': None, 'quantize_and_serve': False, 'rl_quant_profile': None, 'enable_tf32_matmul': False, 'mem_fraction_static': 0.4, 'max_running_requests': None, 'max_queued_requests': None, 'max_total_tokens': None, 'chunked_prefill_size': 4096, 'prefill_decode_interval': 0, 'enable_dynamic_chunking': False, 'max_prefill_tokens': 16384, 'prefill_max_requests': None, 'schedule_policy': 'fcfs', 'enable_priority_scheduling': False, 'disable_priority_preemption': False, 'default_priority_value': None, 'abort_on_priority_when_disabled': False, 'schedule_low_priority_values_first': False, 'priority_scheduling_preemption_threshold': 10, 'retraction_policy': 'length', 'schedule_conservativeness': 1.0, 'page_size': 1, 'c128_page_size': 16, 'swa_full_tokens_ratio': 0.8, 'disable_hybrid_swa_memory': False, 'radix_eviction_policy': 'lru', 'radix_eviction_policy_config': None, 'prefill_only_disable_kv_cache': False, 'disable_radix_cache': False, 'enable_page_major_kv_layout': False, 'enable_unified_memory': False, 'disable_chunked_prefix_cache': False, 'disable_overlap_schedule': False, 'num_continuous_decode_steps': 1, 'scheduler_recv_interval': 1, 'enable_mixed_chunk': False, 'nccl_port': None, 'dist_timeout': None, 'dist_init_addr': None, 'gated_launch_port': None, 'nnodes': 1, 'node_rank': 0, 'tp_size': 1, 'dcp_size': 1, 'pp_size': 1, 'pp_max_micro_batch_size': None, 'pp_async_batch_depth': 0, 'dp_size': 1, 'attn_dp_size': 1, 'load_balance_method': 'round_robin', 'attn_cp_size': 1, 'moe_dp_size': 1, 'dwdp_size': 1, 'dcp_comm_backend': 'ag_rs', 'dcp_replicate_q_proj': None, 'enable_prefill_cp': False, 'cp_strategy': None, 'enable_dsa_cache_layer_split': False, 'enable_cp_decode_attn_tp': False, 'enable_kv_cache_sharding': False, 'enable_dp_attention': False, 'enable_dp_attention_local_control_broadcast': False, 'enable_dp_lm_head': False, 'enable_tp_lm_head_all_to_all': False, 'enable_attn_tp_input_scattered': False, 'enable_shared_experts_attn_tp': False, 'shared_experts_tp_size': None, 'enable_dense_mlp_attn_tp': False, 'enable_layernorm_sp': False, 'disable_attn_tp_gather': False, 'enable_p2p_check': False, 'device': 'mps', 'base_gpu_id': 0, 'gpu_id_step': 1, 'random_seed': 728819419, 'mlx_enable_sampling': False, 'watchdog_timeout': 300, 'soft_watchdog_timeout': None, 'sleep_on_idle': False, 'use_ray': False, 'custom_sigquit_handler': None, 'numa_node': None, 'gc_threshold': None, 'host': '127.0.0.1', 'port': 30000, 'fastapi_root_path': '', 'smg_grpc_mode': False, 'grpc_mode': False, 'grpc_port': None, 'grpc_worker_threads': 4, 'sidecar': None, 'sidecar_args': None, 'skip_server_warmup': False, 'warmups': None, 'enable_http2': False, 'http2_max_concurrent_streams': 200, 'http2_initial_connection_window_size': 1048576, 'ssl_keyfile': None, 'ssl_certfile': None, 'ssl_ca_certs': None, 'ssl_keyfile_password': None, 'enable_ssl_refresh': False, 'api_key': None, 'admin_api_key': None, 'served_model_name': 'mlx-community/Qwen3-0.6B-4bit', 'weight_version': 'default', 'chat_template': None, 'trust_request_chat_template': False, 'hf_chat_template_name': None, 'completion_template': None, 'file_storage_path': None, 'enable_cache_report': True, 'return_input_ids': False, 'return_output_ids': False, 'reasoning_parser': None, 'default_chat_template_kwargs': None, 'strip_thinking_cache': False, 'enable_strict_thinking': False, 'tool_call_parser': None, 'tool_server': None, 'sampling_defaults': 'model', 'asr_max_buffer_seconds': 60, 'asr_max_concurrent_sessions': 32, 'preferred_sampling_params': None, 'allow_auto_truncate': False, 'stream_interval': 1, 'batch_notify_size': 16, 'stream_response_default_include_usage': False, 'incremental_streaming_output': False, 'enable_streaming_session': False, 'enable_session_radix_cache': False, 'log_level': 'info', 'log_level_http': None, 'log_requests': False, 'log_requests_level': 2, 'log_requests_format': 'text', 'log_requests_target': None, 'uvicorn_access_log_exclude_prefixes': [], 'crash_dump_folder': None, 'show_time_cost': False, 'enable_metrics': True, 'smg_http_sidecar_port': None, 'enable_mfu_metrics': False, 'enable_metrics_for_all_schedulers': False, 'load_snapshot_publish_interval': 15, 'tokenizer_metrics_custom_labels_header': 'x-custom-labels', 'tokenizer_metrics_allowed_custom_labels': None, 'extra_metric_labels': None, 'bucket_time_to_first_token': None, 'bucket_inter_token_latency': None, 'bucket_e2e_request_latency': None, 'prompt_tokens_buckets': None, 'generation_tokens_buckets': None, 'gc_warning_threshold_secs': 0.0, 'decode_log_interval': 40, 'enable_request_time_stats_logging': False, 'kv_events_config': None, 'load_publish_endpoint': None, 'enable_forward_pass_metrics': False, 'forward_pass_metrics_worker_id': '', 'forward_pass_metrics_ipc_name': None, 'enable_trace': False, 'trace_modules': 'request', 'otlp_traces_endpoint': 'localhost:4317', 'export_metrics_to_file': False, 'export_metrics_to_file_dir': None, 'stat_loggers': None, 'constrained_json_whitespace_pattern': None, 'constrained_json_disable_any_whitespace': False, 'constrained_json_max_whitespace_cnt': None, 'attention_backend': 'torch_native', 'decode_attention_backend': None, 'enable_lean_attention': None, 'prefill_attention_backend': None, 'sampling_backend': 'pytorch', 'grammar_backend': 'xgrammar', 'radix_cache_backend': None, 'mm_attention_backend': None, 'fp8_gemm_runner_backend': 'auto', 'fp4_gemm_runner_backend': 'auto', 'bf16_gemm_backend': 'auto', 'dsa_prefill_backend': None, 'dsv4_prefill_backend': 'auto', 'dsa_decode_backend': None, 'dsa_paged_mqa_logits_backend': 'auto', 'dsa_topk_backend': 'sgl-kernel', 'disable_flashinfer_autotune': False, 'flashinfer_autotune_skip_ops': None, 'mamba_backend': 'triton', 'cuda_graph_config': {'decode': {'backend': 'disabled', 'max_bs': 32, 'bs': [1, 2, 4, 8, 12, 16, 24, 32], 'tc_compiler': 'eager', 'max_context_size': None, 'full_prefill_max_req': None, 'full_prefill_prefix_chunk_tokens': None, 'max_seq_len': None}, 'prefill': {'backend': 'disabled', 'max_bs': 4096, 'bs': [4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048, 2304, 2560, 2816, 3072, 3328, 3584, 3840, 4096], 'tc_compiler': 'eager', 'max_context_size': None, 'full_prefill_max_req': None, 'full_prefill_prefix_chunk_tokens': None, 'max_seq_len': None}}, 'cuda_graph_backend_decode': None, 'cuda_graph_backend_prefill': None, 'cuda_graph_max_bs_decode': None, 'cuda_graph_max_bs_prefill': None, 'cuda_graph_bs_decode': None, 'cuda_graph_bs_prefill': None, 'cuda_graph_prefill_max_context': None, 'disable_prefill_cuda_graph': False, 'disable_decode_cuda_graph': False, 'disable_cuda_graph': True, 'disable_cuda_graph_padding': False, 'enable_profile_cuda_graph': False, 'enable_cudagraph_gc': False, 'debug_cuda_graph': False, 'enable_layerwise_nvtx_marker': False, 'enable_nccl_nvls': False, 'enable_symm_mem': False, 'triton_attention_reduce_in_fp32': False, 'triton_attention_num_kv_splits': 8, 'triton_attention_split_tile_size': None, 'flashinfer_mla_disable_ragged': False, 'enable_fused_qk_norm_rope': False, 'enable_precise_embedding_interpolation': False, 'enable_fused_moe_sum_all_reduce': False, 'enable_deepseek_v4_fp4_indexer': False, 'disable_custom_all_reduce': False, 'enable_mscclpp': False, 'enable_torch_symm_mem': False, 'enable_scattered_sconv': False, 'pre_warm_nccl': False, 'boundary_reduction': 'ar', 'enable_quant_communications': False, 'enable_flashinfer_allreduce_fusion': False, 'enforce_disable_flashinfer_allreduce_fusion': False, 'flashinfer_allreduce_fusion_backend': None, 'enable_aiter_allreduce_fusion': False, 'enable_torch_compile': False, 'enable_torch_compile_debug_mode': False, 'torch_compile_max_bs': 32, 'speculative_algorithm': None, 'uno_lora_path': None, 'speculative_draft_model_path': None, 'speculative_draft_model_revision': None, 'speculative_draft_load_format': None, 'speculative_num_steps': None, 'speculative_eagle_topk': None, 'speculative_num_draft_tokens': None, 'speculative_dflash_block_size': None, 'speculative_dspark_block_size': None, 'speculative_dspark_sps_table_path': None, 'speculative_dspark_confidence_sts_path': None, 'speculative_dspark_align_verify_tokens_to_graph_tier': False, 'speculative_accept_threshold_single': 1.0, 'speculative_accept_threshold_acc': 1.0, 'speculative_use_rejection_sampling': False, 'speculative_token_map': None, 'speculative_attention_mode': 'prefill', 'speculative_draft_attention_backend': None, 'speculative_dsa_topk_backend': 'sgl-kernel', 'speculative_draft_kv_cache_dtype': None, 'speculative_draft_window_size': None, 'speculative_moe_runner_backend': 'auto', 'speculative_moe_a2a_backend': None, 'speculative_enable_w4a4_mxfp4_megamoe': None, 'speculative_draft_model_quantization': None, '_speculative_draft_quantization_explicitly_set': False, 'speculative_skip_dp_mlp_sync': False, 'enable_multi_layer_eagle': False, 'speculative_adaptive': False, 'speculative_adaptive_config': None, 'decoupled_spec_bind_endpoint': None, 'decoupled_spec_connect_endpoints': None, 'decoupled_spec_rank': None, 'decoupled_spec_role': 'null', 'spec_trace_dir': None, 'speculative_ngram_min_bfs_breadth': 1, 'speculative_ngram_max_bfs_breadth': 10, 'speculative_ngram_match_type': 'BFS', 'speculative_ngram_max_trie_depth': 18, 'speculative_ngram_capacity': 10000000, 'speculative_ngram_external_corpus_path': None, 'speculative_ngram_external_sam_budget': 0, 'speculative_ngram_external_corpus_max_tokens': 10000000, 'ep_size': 1, 'moe_a2a_backend': 'none', 'enable_w4a4_mxfp4_megamoe': False, 'deepep_v2_mode': 'direct', 'moe_runner_backend': 'auto', 'flashinfer_mxfp4_moe_precision': 'default', 'deepep_mode': 'auto', 'fuseep_mode': 2, 'deepep_dispatcher_output_dtype': 'auto', 'ep_num_redundant_experts': 0, 'ep_dispatch_algorithm': None, 'init_expert_location': 'trivial', 'enable_eplb': False, 'eplb_algorithm': 'auto', 'eplb_rebalance_num_iterations': 1000, 'eplb_rebalance_layers_per_chunk': None, 'eplb_min_rebalancing_utilization_threshold': 1.0, 'expert_distribution_recorder_mode': None, 'expert_distribution_recorder_buffer_size': 1000, 'expert_balancedness_report_mode': 'off', 'deepep_config': None, 'moe_dense_tp_size': None, 'elastic_ep_backend': None, 'enable_elastic_expert_backup': False, 'mooncake_ib_device': None, 'enable_waterfill': False, 'ep_join_mode': None, 'ep_join_rank_offset': 0, 'elastic_ep_initial_size': None, 'max_ep_size': None, 'elastic_ep_scale_timeout': 600, 'disable_flashinfer_cutlass_moe_fp4_allgather': False, 'disable_shared_experts_fusion': False, 'enforce_shared_experts_fusion': False, 'max_mamba_cache_size': None, 'mamba_ssm_dtype': None, 'mamba_max_states_per_path': -1, 'enable_mamba_cache_stochastic_rounding': False, 'mamba_cache_philox_rounds': 0, 'mamba_full_memory_ratio': 0.9, 'mamba_radix_cache_strategy': 'auto', 'uses_mamba_radix_cache': False, 'mamba_track_interval': 256, 'enable_int8_mamba_checkpoint': False, 'int8_mamba_ckpt_size': None, 'linear_attn_backend': 'triton', 'linear_attn_decode_backend': None, 'linear_attn_prefill_backend': None, 'linear_attn_verify_backend': None, 'enable_linear_replayssm': False, 'linear_replayssm_cache_len': 16, 'enable_linear_replayssm_spec': False, 'enable_hierarchical_cache': False, 'hicache_host_memory_mode': 'cache', 'hicache_ratio': 2.0, 'hicache_size': 0, 'hicache_write_policy': 'write_through', 'hicache_io_backend': 'kernel', 'hicache_mem_layout': 'page_first', 'hicache_storage_backend': None, 'hicache_storage_prefetch_policy': 'timeout', 'hicache_storage_backend_extra_config': None, 'hicache_storage_prefetch_retry_poll_interval': 8, 'hicache_storage_prefetch_retry_max_attempts': 8, 'enable_unified_cache_external_linker': False, 'unified_cache_external_linker_backend': 'mooncake', 'enable_hisparse': False, 'hisparse_config': None, 'enable_broadcast_mm_inputs_process': False, 'enable_prefix_mm_cache': False, 'mm_enable_dp_encoder': False, 'mm_process_config': {}, 'mm_processor_worker_num': 0, 'mm_io_worker_num': 0, 'allowed_media_domains': [], 'media_url_max_file_size_mb': 64, 'mm_preprocess_cache_size_mb': None, 'trust_mm_content_hashes': False, 'limit_mm_data_per_request': None, 'enable_mm_global_cache': False, 'image_processor_backend': 'auto', 'mm_global_cache_backend': 'mooncake', 'disable_fast_image_processor': False, 'mm_feature_transport': 'cpu', 'enable_lora': None, 'enable_lora_overlap_loading': None, 'max_lora_rank': None, 'lora_target_modules': None, 'lora_paths': None, 'max_loaded_loras': None, 'max_loras_per_batch': 8, 'lora_eviction_policy': 'lru', 'lora_backend': 'csgmv', 'max_lora_chunk_size': 16, 'experts_shared_outer_loras': None, 'lora_use_virtual_experts': False, 'lora_strict_loading': False, 'lora_drain_wait_threshold': 0.0, 'enable_two_batch_overlap': False, 'enable_single_batch_overlap': False, 'tbo_token_distribution_threshold': 0.48, 'cpu_offload_gb': 0, 'offload_group_size': -1, 'offload_num_in_group': 1, 'offload_prefetch_step': 1, 'offload_mode': 'cpu', 'enable_lmcache': False, 'lmcache_config_file': None, 'enable_flexkv': False, 'flexkv_config_file': None, 'kt_weight_path': None, 'kt_method': 'AMXINT4', 'kt_cpuinfer': None, 'kt_threadpool_count': 2, 'kt_num_gpu_experts': None, 'kt_max_deferred_experts_per_token': None, 'dllm_algorithm': None, 'dllm_algorithm_config': None, 'dllm_fdfo': True, 'disaggregation_mode': 'null', 'disaggregation_transfer_backend': 'mooncake', 'disaggregation_bootstrap_port': 8998, 'disaggregation_ib_device': None, 'disaggregation_decode_enable_radix_cache': False, 'disaggregation_decode_enable_offload_kvcache': False, 'disaggregation_decode_retraction_backup': None, 'num_reserved_decode_tokens': 512, 'disaggregation_decode_extra_slots': None, 'disaggregation_decode_polling_interval': 1, 'optimistic_prefill_attempts': 0, 'encoder_only': False, 'language_only': False, 'language_model_only': False, 'encoder_transfer_backend': 'zmq_to_scheduler', 'encoder_urls': [], 'encoder_bootstrap_port': 8997, 'encoder_register_urls': [], 'enable_adaptive_dispatch_to_encoder': False, 'enable_pdmux': False, 'pdmux_config_path': None, 'sm_group_num': 8, 'startup_weight_load_mode': 'serial', 'custom_weight_loader': [], 'weight_loader_disable_mmap': False, 'weight_loader_prefetch_checkpoints': False, 'weight_loader_prefetch_num_threads': 4, 'weight_loader_drop_cache_after_load': False, 'remote_instance_weight_loader_seed_instance_ip': None, 'remote_instance_weight_loader_seed_instance_service_port': None, 'remote_instance_weight_loader_send_weights_group_ports': None, 'remote_instance_weight_loader_backend': 'nccl', 'remote_instance_weight_loader_start_seed_via_transfer_engine': False, 'engine_info_bootstrap_port': 6789, 'modelexpress_config': None, 'download_dir': None, 'model_checksum': None, 'delete_ckpt_after_loading': False, 'decrypted_config_file': None, 'decrypted_draft_config_file': None, 'checkpoint_engine_wait_weights_before_ready': False, 'enable_prefill_delayer': False, 'prefill_delayer_max_delay_passes': 30, 'prefill_delayer_token_usage_low_watermark': None, 'prefill_delayer_forward_passes_buckets': None, 'prefill_delayer_wait_seconds_buckets': None, 'prefill_delayer_queue_min_ratio': None, 'prefill_delayer_max_delay_ms': None, 'min_free_slots_delay': None, 'enable_deterministic_inference': False, 'rl_on_policy_target': None, 'kv_canary': 'none', 'kv_canary_real_data': 'none', 'kv_canary_sweep_interval': 0, 'enable_dynamic_batch_tokenizer': False, 'dynamic_batch_tokenizer_batch_size': 32, 'dynamic_batch_tokenizer_batch_timeout': 0.002, 'enable_tokenizer_batch_encode': False, 'disable_tokenizer_batch_decode': False, 'debug_tensor_dump_output_folder': None, 'debug_tensor_dump_layers': None, 'debug_tensor_dump_input_file': None, 'enable_memory_saver': False, 'enable_weights_cpu_backup': False, 'enable_draft_weights_cpu_backup': False, 'enable_custom_logit_processor': False, 'enable_return_hidden_states': False, 'return_hidden_states_mode': None, 'enable_return_routed_experts': False, 'enable_return_indexer_topk': False, 'sampling_mask_max_tokens': 4096, 'disable_outlines_disk_cache': False, 'enable_mis': False, 'weight_cache_mode': 'off', 'weight_cache_socket': None, 'weight_cache_timeout': 1800, 'forward_hooks': None, 'msprobe_dump_config': None, 'ple_offload_embedding': None, 'ple_offload_backend': 'pinned', 'ple_offload_dir': None, 'disaggregation_enable_kv_checksum': False, 'enable_pd_role_switch': False, 'speculative_domino_candidate_pool_size': 2048, 'flashinfer_a2a_dispatch_type': None, 'cuda_graph_max_seq_len_prefill': None, 'prefill_kv_cache_dequant_dtype': 'auto', 'dsv4_attn_backend': 'auto', 'otlp_service_name': None, '_radix_eviction_policy_explicitly_set': False, 'hicache_host_memory_fraction': 0.8, 'enable_linker_mla_dedup': False, '_swa_full_tokens_ratio_explicitly_set': False, 'swa_prefix_tails': None, 'enable_encoder_swa_bounded_replay': False, 'enable_decoder_swa_bounded_replay': False, 'enable_response_store': False, 'disaggregation_decode_host_receive_threshold': 1.0, 'qsa_indexer_dtype': 'auto', 'disaggregation_decode_allocation_policy': 'early', 'speculative_boundary_reduction': 'ar', 'speculative_use_block_verification': False, 'speculative_draft_sink_size': None, 'disable_aiter_allreduce_fusion_in_prefill': False, 'disable_aiter_allreduce_fusion_in_decode': False, 'dsa_triton_union': 0, 'enable_dsa_fused_indexer': None, 'enable_cp_tp_group_sharing': False, 'enable_watermark': False, 'watermark_config': '<redacted>', 'watermark_key': '<redacted>', 'watermark_key_b': '<redacted>', 'watermark_context_window': 4, 'watermark_mixing_probability': 0.5, 'watermark_max_probability': 1.0, 'watermark_default_enabled': False, 'watermark_enforce_all': False, 'grpc_response_timeout_secs': 300}
[2026-10-16 13:57:55] mlx_q4 quantization is not fully optimized yet. The speed can be slower than non-quantized models.
[2026-10-16 13:57:55] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit "HTTP/1.1 200 OK"
[2026-10-16 13:57:55] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
[2026-10-16 13:57:55] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
[2026-10-16 13:57:56] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit "HTTP/1.1 200 OK"
[2026-10-16 13:57:56] Using default HuggingFace chat template with detected content format: string
[2026-10-16 13:57:56] Auto-detected template features: reasoning_config=ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None), reasoning_parser=qwen3, tool_call_parser=qwen
/private/tmp/sglang-research/sglang-v0.5.22/python/sglang/kernels/ops/attention/fla/utils.py:223: UserWarning: Triton is not supported on current platform, roll back to CPU.
  warnings.warn(
/private/tmp/sglang-research/sglang-v0.5.22/python/sglang/srt/layers/quantization/awq/awq.py:53: UserWarning: Only CUDA, HIP and XPU support AWQ currently.
  warnings.warn(f"Only CUDA, HIP and XPU support AWQ currently.")
/private/tmp/sglang-research/sglang-v0.5.22/python/sglang/srt/layers/quantization/awq/awq.py:53: UserWarning: Only CUDA, HIP and XPU support AWQ currently.
  warnings.warn(f"Only CUDA, HIP and XPU support AWQ currently.")
/private/tmp/sglang-research/sglang-v0.5.22/python/sglang/srt/layers/quantization/gguf.py:69: UserWarning: Only CUDA, MUSA and NPU support GGUF quantization currently.
  warnings.warn(f"Only CUDA, MUSA and NPU support GGUF quantization currently.")
/private/tmp/sglang-research/sglang-v0.5.22/python/sglang/srt/layers/quantization/gguf.py:69: UserWarning: Only CUDA, MUSA and NPU support GGUF quantization currently.
  warnings.warn(f"Only CUDA, MUSA and NPU support GGUF quantization currently.")
[2026-10-16 13:58:00] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit "HTTP/1.1 200 OK"
[2026-10-16 13:58:00] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit "HTTP/1.1 200 OK"
[2026-10-16 13:58:00] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
[2026-10-16 13:58:00] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit "HTTP/1.1 200 OK"
[2026-10-16 13:58:00] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
[2026-10-16 13:58:00] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit "HTTP/1.1 200 OK"
[2026-10-16 13:58:00] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit "HTTP/1.1 200 OK"
[2026-10-16 13:58:00] mlx_q4 quantization is not fully optimized yet. The speed can be slower than non-quantized models.
[2026-10-16 13:58:00] Init parallel begin.
[2026-10-16 13:58:01] Init parallel ends. elapsed=0.08 s
[2026-10-16 13:58:01] load snapshot writer init failed: [Errno 2] No such file or directory: '/dev/shm/sglang_loads_96164160c86b_990a6799.shm'
[2026-10-16 13:58:01] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit "HTTP/1.1 200 OK"
[2026-10-16 13:58:01] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit "HTTP/1.1 200 OK"
[2026-10-16 13:58:01] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
[2026-10-16 13:58:01] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
[2026-10-16 13:58:02] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit "HTTP/1.1 200 OK"
[2026-10-16 13:58:02] mlx_q4 quantization is not fully optimized yet. The speed can be slower than non-quantized models.
[2026-10-16 13:58:02] Initializing MlxModelRunner for end-to-end MLX inference
[2026-10-16 13:58:02] Loading MLX model: mlx-community/Qwen3-0.6B-4bit
[2026-10-16 13:58:03] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit/revision/main "HTTP/1.1 200 OK"

Fetching 9 files:   0%|          | 0/9 [00:00<?, ?it/s]
Fetching 9 files: 100%|██████████| 9/9 [00:00<00:00, 3882.82it/s]
[2026-10-16 13:58:03] MLX model loaded in 0.97s
[2026-10-16 13:58:03] Wired memory limit set to 51.8 GB
[2026-10-16 13:58:03] Auto-sized attention KV pool: sys_available=48.49 GB, mlx_limit=51.8 GB, mlx_used=0.31 GB, kv_budget=19.39 GB, bytes_per_slot=114688, pool_size=181574
[2026-10-16 13:58:03] Memory baseline taken. mem usage=0.00 GB
[2026-10-16 13:58:03] MLX stub: skipping PyTorch model weight loading (inference runs through MLX)
[2026-10-16 13:58:04] MLX stub: initialized minimal pools (max_total_num_tokens=181574, max_running_requests=4096, zero GPU KV cache allocation)
[2026-10-16 13:58:04] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit "HTTP/1.1 200 OK"
[2026-10-16 13:58:04] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
[2026-10-16 13:58:04] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
[2026-10-16 13:58:05] HTTP Request: GET https://huggingface.co/api/models/mlx-community/Qwen3-0.6B-4bit "HTTP/1.1 200 OK"
[2026-10-16 13:58:05] Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' (e.g. via --cuda-graph-backend-prefill=disabled or auto-disable rules).
[2026-10-16 13:58:05] max_total_num_tokens=181574, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=4096, context_len=40960, available_gpu_mem=47.96 GB
[2026-10-16 13:58:05] Cannot detect CUDA architecture.
[2026-10-16 13:58:05] Ignore import error when loading sglang.srt.models.bailing_moe_linear: No module named 'vllm'
[2026-10-16 13:58:05] Ignore import error when loading sglang.srt.models.bailing_moe_nextn: No module named 'vllm'
[2026-10-16 13:58:06] Ignore import error when loading sglang.srt.models.falcon_mamba: cannot import name 'causal_conv1d_fn' from 'sglang.srt.layers.attention.mamba.mamba' (/private/tmp/sglang-research/sglang-v0.5.22/python/sglang/srt/layers/attention/mamba/mamba.py)
[2026-10-16 13:58:06] Ignore import error when loading sglang.srt.models.inkling: No module named 'cutlass'
[2026-10-16 13:58:06] Ignore import error when loading sglang.srt.models.mamba: cannot import name 'causal_conv1d_fn' from 'sglang.srt.layers.attention.mamba.mamba' (/private/tmp/sglang-research/sglang-v0.5.22/python/sglang/srt/layers/attention/mamba/mamba.py)
[2026-10-16 13:58:06] Using the Python TreeCore: the Rust TreeCore supports Linux only
[2026-10-16 13:58:06] Init Unified Radix Cache. Components: (<ComponentType.FULL: 0>,). Tree Core: UnifiedTreeCore
[2026-10-16 13:58:06] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=False hybrid_ssm=False hicache_attached=False
[2026-10-16 13:58:06] Engine startup timings (s): load_weight=0.00, kv_cache_allocation=0.00, scheduler_e2e=7.01, cuda_graph={prefill=0.00, decode=0.00, target_verify=0.00, draft_prefill=0.00, draft_decode=0.00, draft_extend=0.00}, tokenizer_e2e=11.92
[2026-10-16 13:58:06] INFO:     Started server process [5782]
[2026-10-16 13:58:06] INFO:     Waiting for application startup.
[2026-10-16 13:58:06] INFO:     Application startup complete.
[2026-10-16 13:58:07] INFO:     Uvicorn running on http://127.0.0.1:30000 (Press CTRL+C to quit)
[2026-10-16 13:58:08] INFO:     127.0.0.1:63780 - "GET /model_info HTTP/1.1" 200 OK
[2026-10-16 13:58:08] MlxAttentionKVPool: 181575 slots x 28 layers x 8 heads x 128 dim, dtype=mlx.core.bfloat16, ~19859.8 MB
[2026-10-16 13:58:08] Attention KV pool initialized: pool_size=181574 (buffer size 181575 incl. padding slot 0), 28 full-attention layers (0 sliding-window layers stay per-request), 8 kv_heads, 128 head_dim
[2026-10-16 13:58:08] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 3.85
[2026-10-16 13:58:08] INFO:     127.0.0.1:63781 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:58:08] INFO:     127.0.0.1:63785 - "POST /freeze_gc HTTP/1.1" 200 OK
[2026-10-16 13:58:08] The server is fired up and ready to roll!
[2026-10-16 13:58:08] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 2.01
[2026-10-16 13:58:09] INFO:     127.0.0.1:63787 - "GET /health HTTP/1.1" 200 OK
[2026-10-16 13:58:10] Prefill batch, #new-seq: 1, #new-token: 3, #cached-token: 2, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 1.55
[2026-10-16 13:58:10] INFO:     127.0.0.1:63795 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:58:10] Prefill batch, #new-seq: 1, #new-token: 21, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 258.16
[2026-10-16 13:58:10] INFO:     127.0.0.1:63797 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-16 13:58:10] Prefill batch, #new-seq: 1, #new-token: 21, #cached-token: 1, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 130.47
[2026-10-16 13:58:10] INFO:     127.0.0.1:63798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-16 13:58:10] Decode batch, #running-req: 1, #token: 0, token usage: 0.00, cuda graph: False, gen throughput (token/s): 9.33, #queue-req: 0
[2026-10-16 13:58:11] Prefill batch, #new-seq: 1, #new-token: 1176, #cached-token: 3, token usage: 0.01, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 2952.78
[2026-10-16 13:58:11] INFO:     127.0.0.1:63799 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-16 13:58:12] Prefill batch, #new-seq: 1, #new-token: 2996, #cached-token: 3, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 2990.72
[2026-10-16 13:58:12] INFO:     127.0.0.1:63801 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-16 13:58:12] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 2998, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 7.09
[2026-10-16 13:58:12] INFO:     127.0.0.1:63804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-16 13:58:12] Decode batch, #running-req: 1, #token: 3000, token usage: 0.02, cuda graph: False, gen throughput (token/s): 26.10, #queue-req: 0
[2026-10-16 13:58:12] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 2998, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 8.42
[2026-10-16 13:58:12] INFO:     127.0.0.1:63805 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-16 13:58:12] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 2998, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 6.99
[2026-10-16 13:58:12] INFO:     127.0.0.1:63806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-16 13:58:12] Decode batch, #running-req: 1, #token: 0, token usage: 0.00, cuda graph: False, gen throughput (token/s): 115.64, #queue-req: 0
[2026-10-16 13:58:12] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 2998, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 5.48
[2026-10-16 13:58:12] INFO:     127.0.0.1:63807 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-16 13:58:14] Prefill batch, #new-seq: 1, #new-token: 3031, #cached-token: 3, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 1937.91
[2026-10-16 13:58:14] INFO:     127.0.0.1:63809 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-16 13:58:16] Prefill batch, #new-seq: 1, #new-token: 3027, #cached-token: 5, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 1301.93
[2026-10-16 13:58:16] INFO:     127.0.0.1:63812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-16 13:58:16] Decode batch, #running-req: 1, #token: 3033, token usage: 0.02, cuda graph: False, gen throughput (token/s): 10.03, #queue-req: 0
[2026-10-16 13:58:17] Prefill batch, #new-seq: 1, #new-token: 3025, #cached-token: 5, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 2255.03
[2026-10-16 13:58:18] INFO:     127.0.0.1:63816 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-16 13:58:19] Prefill batch, #new-seq: 1, #new-token: 3025, #cached-token: 6, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 2559.49
[2026-10-16 13:58:19] INFO:     127.0.0.1:63819 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-16 13:58:19] Decode batch, #running-req: 1, #token: 0, token usage: 0.00, cuda graph: False, gen throughput (token/s): 15.64, #queue-req: 0
[2026-10-16 13:58:20] Prefill batch, #new-seq: 1, #new-token: 3028, #cached-token: 6, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 2404.31
[2026-10-16 13:58:20] INFO:     127.0.0.1:63821 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-16 13:58:20] Prefill batch, #new-seq: 1, #new-token: 45, #cached-token: 2983, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 225.19
[2026-10-16 13:58:20] INFO:     127.0.0.1:63826 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-16 13:58:20] Prefill batch, #new-seq: 1, #new-token: 48, #cached-token: 2985, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 271.94
[2026-10-16 13:58:20] INFO:     127.0.0.1:63827 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-16 13:58:20] Decode batch, #running-req: 1, #token: 3034, token usage: 0.02, cuda graph: False, gen throughput (token/s): 25.03, #queue-req: 0
[2026-10-16 13:58:20] Prefill batch, #new-seq: 1, #new-token: 45, #cached-token: 2986, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 266.77
[2026-10-16 13:58:20] INFO:     127.0.0.1:63828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-16 13:58:21] Prefill batch, #new-seq: 1, #new-token: 48, #cached-token: 2985, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 231.22
[2026-10-16 13:58:21] INFO:     127.0.0.1:63830 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-10-16 13:58:21] Decode batch, #running-req: 1, #token: 0, token usage: 0.00, cuda graph: False, gen throughput (token/s): 96.91, #queue-req: 0
[2026-10-16 13:58:21] Prefill batch, #new-seq: 1, #new-token: 43, #cached-token: 2986, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 266.46
[2026-10-16 13:58:21] INFO:     127.0.0.1:63831 - "POST /v1/chat/completions HTTP/1.1" 200 OK
/private/tmp/sglang-research/sglang-v0.5.22/python/sglang/srt/entrypoints/http_server.py:476: FastAPIDeprecationWarning: ORJSONResponse is deprecated, FastAPI now serializes data directly to JSON bytes via Pydantic when a return type or response model is set, which is faster and doesn't need a custom response class. Read more in the FastAPI docs: https://fastapi.tiangolo.com/advanced/custom-response/#orjson-or-response-model and https://fastapi.tiangolo.com/tutorial/response-model/
  return await original_handler(ORJSONRequest(request.scope, request.receive))
[2026-10-16 13:58:26] INFO:     127.0.0.1:63838 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-16 13:58:26] INFO:     127.0.0.1:63839 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-16 13:58:28] INFO:     127.0.0.1:63845 - "GET /model_info HTTP/1.1" 200 OK
[2026-10-16 13:58:32] INFO:     127.0.0.1:63855 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:58:33] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 42.60
[2026-10-16 13:58:33] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 3.28, #queue-req: 0
[2026-10-16 13:58:34] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 1.12
[2026-10-16 13:58:34] INFO:     127.0.0.1:63859 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:58:34] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 511, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 2.77
[2026-10-16 13:58:34] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 31.72, #queue-req: 0
[2026-10-16 13:58:34] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 271.55, #queue-req: 0
[2026-10-16 13:58:35] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 252.78, #queue-req: 0
[2026-10-16 13:58:35] INFO:     127.0.0.1:63861 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:58:35] Prefill batch, #new-seq: 1, #new-token: 256, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 458.27
[2026-10-16 13:58:35] INFO:     127.0.0.1:63858 - "GET /health HTTP/1.1" 200 OK
[2026-10-16 13:58:35] Decode batch, #running-req: 1, #token: 257, token usage: 0.00, cuda graph: False, gen throughput (token/s): 180.16, #queue-req: 0
[2026-10-16 13:58:35] Decode batch, #running-req: 1, #token: 257, token usage: 0.00, cuda graph: False, gen throughput (token/s): 268.75, #queue-req: 0
[2026-10-16 13:58:35] Decode batch, #running-req: 1, #token: 257, token usage: 0.00, cuda graph: False, gen throughput (token/s): 286.90, #queue-req: 0
[2026-10-16 13:58:35] INFO:     127.0.0.1:63864 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:58:35] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 864.58
[2026-10-16 13:58:35] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 137.59, #queue-req: 0
[2026-10-16 13:58:35] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 280.05, #queue-req: 0
[2026-10-16 13:58:36] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 271.22, #queue-req: 0
[2026-10-16 13:58:36] INFO:     127.0.0.1:63871 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:58:36] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 822.54
[2026-10-16 13:58:36] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 129.35, #queue-req: 0
[2026-10-16 13:58:36] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 273.89, #queue-req: 0
[2026-10-16 13:58:36] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 246.22, #queue-req: 0
[2026-10-16 13:58:36] Decode batch, #running-req: 1, #token: 0, token usage: 0.00, cuda graph: False, gen throughput (token/s): 256.38, #queue-req: 0
[2026-10-16 13:58:36] INFO:     127.0.0.1:63872 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:58:37] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 776.47
[2026-10-16 13:58:37] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 131.99, #queue-req: 0
[2026-10-16 13:58:37] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 232.18, #queue-req: 0
[2026-10-16 13:58:37] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 218.21, #queue-req: 0
[2026-10-16 13:58:37] INFO:     127.0.0.1:63874 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:58:37] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 1, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 641.23
[2026-10-16 13:58:38] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 90.37, #queue-req: 0
[2026-10-16 13:58:38] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 183.17, #queue-req: 0
[2026-10-16 13:58:38] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 210.14, #queue-req: 0
[2026-10-16 13:58:38] INFO:     127.0.0.1:63878 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:58:38] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 614.18
[2026-10-16 13:58:38] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 96.96, #queue-req: 0
[2026-10-16 13:58:39] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 244.27, #queue-req: 0
[2026-10-16 13:58:39] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 263.18, #queue-req: 0
[2026-10-16 13:58:39] INFO:     127.0.0.1:63880 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:58:39] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 699.51
[2026-10-16 13:58:39] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 112.06, #queue-req: 0
[2026-10-16 13:58:39] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 215.77, #queue-req: 0
[2026-10-16 13:58:39] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 209.74, #queue-req: 0
[2026-10-16 13:58:40] INFO:     127.0.0.1:63882 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:58:40] Prefill batch, #new-seq: 1, #new-token: 496, #cached-token: 1, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 560.00
[2026-10-16 13:58:40] Decode batch, #running-req: 1, #token: 498, token usage: 0.00, cuda graph: False, gen throughput (token/s): 86.20, #queue-req: 0
[2026-10-16 13:58:40] Decode batch, #running-req: 1, #token: 498, token usage: 0.00, cuda graph: False, gen throughput (token/s): 212.04, #queue-req: 0
[2026-10-16 13:58:40] Decode batch, #running-req: 1, #token: 498, token usage: 0.00, cuda graph: False, gen throughput (token/s): 195.26, #queue-req: 0
[2026-10-16 13:58:40] INFO:     127.0.0.1:63884 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:58:40] Decode batch, #running-req: 1, #token: 0, token usage: 0.00, cuda graph: False, gen throughput (token/s): 201.49, #queue-req: 0
[2026-10-16 13:58:41] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 607.20
[2026-10-16 13:58:41] Decode batch, #running-req: 1, #token: 512, token usage: 0.00, cuda graph: False, gen throughput (token/s): 98.91, #queue-req: 0
[2026-10-16 13:58:41] Decode batch, #running-req: 1, #token: 512, token usage: 0.00, cuda graph: False, gen throughput (token/s): 193.14, #queue-req: 0
[2026-10-16 13:58:41] Decode batch, #running-req: 1, #token: 512, token usage: 0.00, cuda graph: False, gen throughput (token/s): 198.09, #queue-req: 0
[2026-10-16 13:58:41] INFO:     127.0.0.1:63886 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:58:41] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 612.74
[2026-10-16 13:58:42] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 99.71, #queue-req: 0
[2026-10-16 13:58:42] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 209.32, #queue-req: 0
[2026-10-16 13:58:42] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 211.25, #queue-req: 0
[2026-10-16 13:58:42] INFO:     127.0.0.1:63888 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:58:42] Prefill batch, #new-seq: 1, #new-token: 509, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 638.69
[2026-10-16 13:58:42] Decode batch, #running-req: 1, #token: 510, token usage: 0.00, cuda graph: False, gen throughput (token/s): 108.14, #queue-req: 0
[2026-10-16 13:58:43] Decode batch, #running-req: 1, #token: 510, token usage: 0.00, cuda graph: False, gen throughput (token/s): 224.82, #queue-req: 0
[2026-10-16 13:58:43] Decode batch, #running-req: 1, #token: 510, token usage: 0.00, cuda graph: False, gen throughput (token/s): 226.05, #queue-req: 0
[2026-10-16 13:58:43] INFO:     127.0.0.1:63890 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:58:43] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 689.10
[2026-10-16 13:58:43] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 117.57, #queue-req: 0
[2026-10-16 13:58:43] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 259.65, #queue-req: 0
[2026-10-16 13:58:43] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 259.41, #queue-req: 0
[2026-10-16 13:58:44] INFO:     127.0.0.1:63892 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:58:44] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 772.29
[2026-10-16 13:58:44] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 123.55, #queue-req: 0
[2026-10-16 13:58:44] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 258.02, #queue-req: 0
[2026-10-16 13:58:44] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 254.86, #queue-req: 0
[2026-10-16 13:58:44] INFO:     127.0.0.1:63895 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:58:44] Decode batch, #running-req: 1, #token: 0, token usage: 0.00, cuda graph: False, gen throughput (token/s): 254.90, #queue-req: 0
[2026-10-16 13:58:44] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 767.46
[2026-10-16 13:58:45] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 124.40, #queue-req: 0
[2026-10-16 13:58:45] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 267.15, #queue-req: 0
[2026-10-16 13:58:45] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 269.66, #queue-req: 0
[2026-10-16 13:58:45] INFO:     127.0.0.1:63897 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:58:45] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 1, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 664.28
[2026-10-16 13:58:45] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 93.91, #queue-req: 0
[2026-10-16 13:58:45] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 188.91, #queue-req: 0
[2026-10-16 13:58:46] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 252.53, #queue-req: 0
[2026-10-16 13:58:46] INFO:     127.0.0.1:63901 - "GET /server_info HTTP/1.1" 200 OK
[2026-10-16 13:58:46] INFO:     127.0.0.1:63902 - "GET /server_info HTTP/1.1" 200 OK
[2026-10-16 13:58:52] INFO:     127.0.0.1:63914 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-16 13:58:52] INFO:     127.0.0.1:63915 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-16 13:58:53] INFO:     127.0.0.1:63918 - "GET /model_info HTTP/1.1" 200 OK
[2026-10-16 13:58:58] INFO:     127.0.0.1:63926 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:58:59] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 511, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.07
[2026-10-16 13:58:59] Decode batch, #running-req: 1, #token: 513, token usage: 0.00, cuda graph: False, gen throughput (token/s): 3.06, #queue-req: 0
[2026-10-16 13:59:00] INFO:     127.0.0.1:63933 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:00] INFO:     127.0.0.1:63934 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:00] INFO:     127.0.0.1:63935 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:00] INFO:     127.0.0.1:63936 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:00] INFO:     127.0.0.1:63937 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:00] INFO:     127.0.0.1:63938 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:00] INFO:     127.0.0.1:63939 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:00] INFO:     127.0.0.1:63940 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:00] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 511, token usage: 0.00, #running-req: 0, #queue-req: 7, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.84
[2026-10-16 13:59:00] Prefill batch, #new-seq: 7, #new-token: 7, #cached-token: 3321, token usage: 0.02, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 40.93
[2026-10-16 13:59:01] Decode batch, #running-req: 8, #token: 3847, token usage: 0.02, cuda graph: False, gen throughput (token/s): 143.83, #queue-req: 0
[2026-10-16 13:59:01] Decode batch, #running-req: 8, #token: 3847, token usage: 0.02, cuda graph: False, gen throughput (token/s): 457.20, #queue-req: 0
[2026-10-16 13:59:02] Decode batch, #running-req: 8, #token: 3847, token usage: 0.02, cuda graph: False, gen throughput (token/s): 400.03, #queue-req: 0
[2026-10-16 13:59:03] INFO:     127.0.0.1:63944 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:03] INFO:     127.0.0.1:63945 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:03] INFO:     127.0.0.1:63946 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:03] INFO:     127.0.0.1:63947 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:03] INFO:     127.0.0.1:63948 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:03] INFO:     127.0.0.1:63950 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:03] INFO:     127.0.0.1:63951 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:03] INFO:     127.0.0.1:63952 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:03] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 496, token usage: 0.00, #running-req: 0, #queue-req: 7, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.35
[2026-10-16 13:59:03] Prefill batch, #new-seq: 7, #new-token: 7, #cached-token: 3573, token usage: 0.02, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 12.23
[2026-10-16 13:59:04] Decode batch, #running-req: 8, #token: 4084, token usage: 0.02, cuda graph: False, gen throughput (token/s): 165.17, #queue-req: 0
[2026-10-16 13:59:05] Decode batch, #running-req: 8, #token: 4084, token usage: 0.02, cuda graph: False, gen throughput (token/s): 386.61, #queue-req: 0
[2026-10-16 13:59:06] Decode batch, #running-req: 8, #token: 4084, token usage: 0.02, cuda graph: False, gen throughput (token/s): 334.69, #queue-req: 0
[2026-10-16 13:59:06] INFO:     127.0.0.1:63958 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:06] INFO:     127.0.0.1:63959 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:06] INFO:     127.0.0.1:63960 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:06] INFO:     127.0.0.1:63961 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:06] INFO:     127.0.0.1:63962 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:06] INFO:     127.0.0.1:63963 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:06] INFO:     127.0.0.1:63964 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:06] INFO:     127.0.0.1:63965 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:07] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 1, token usage: 0.00, #running-req: 0, #queue-req: 7, #pending-token: 0, cuda graph: False, input throughput (token/s): 155.03
[2026-10-16 13:59:09] Prefill batch, #new-seq: 7, #new-token: 3553, #cached-token: 2, token usage: 0.02, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 1914.13
[2026-10-16 13:59:09] Decode batch, #running-req: 8, #token: 4075, token usage: 0.02, cuda graph: False, gen throughput (token/s): 98.37, #queue-req: 0
[2026-10-16 13:59:10] Decode batch, #running-req: 8, #token: 4075, token usage: 0.02, cuda graph: False, gen throughput (token/s): 382.25, #queue-req: 0
[2026-10-16 13:59:11] Decode batch, #running-req: 8, #token: 4075, token usage: 0.02, cuda graph: False, gen throughput (token/s): 399.97, #queue-req: 0
[2026-10-16 13:59:11] INFO:     127.0.0.1:63975 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:11] INFO:     127.0.0.1:63976 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:11] INFO:     127.0.0.1:63977 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:11] INFO:     127.0.0.1:63978 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:11] INFO:     127.0.0.1:63979 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:11] INFO:     127.0.0.1:63980 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:11] INFO:     127.0.0.1:63981 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:11] INFO:     127.0.0.1:63982 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:11] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 7, #pending-token: 0, cuda graph: False, input throughput (token/s): 175.43
[2026-10-16 13:59:13] Prefill batch, #new-seq: 7, #new-token: 3572, #cached-token: 8, token usage: 0.02, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 2511.60
[2026-10-16 13:59:13] Decode batch, #running-req: 8, #token: 4100, token usage: 0.02, cuda graph: False, gen throughput (token/s): 126.94, #queue-req: 0
[2026-10-16 13:59:14] Decode batch, #running-req: 8, #token: 4100, token usage: 0.02, cuda graph: False, gen throughput (token/s): 352.46, #queue-req: 0
[2026-10-16 13:59:15] Decode batch, #running-req: 8, #token: 4100, token usage: 0.02, cuda graph: False, gen throughput (token/s): 387.44, #queue-req: 0
[2026-10-16 13:59:16] Decode batch, #running-req: 8, #token: 0, token usage: 0.00, cuda graph: False, gen throughput (token/s): 382.81, #queue-req: 0
[2026-10-16 13:59:16] INFO:     127.0.0.1:63990 - "GET /server_info HTTP/1.1" 200 OK
[2026-10-16 13:59:16] INFO:     127.0.0.1:63991 - "GET /server_info HTTP/1.1" 200 OK
[2026-10-16 13:59:23] INFO:     127.0.0.1:64000 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-16 13:59:23] INFO:     127.0.0.1:64001 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-16 13:59:25] INFO:     127.0.0.1:64006 - "GET /model_info HTTP/1.1" 200 OK
[2026-10-16 13:59:29] INFO:     127.0.0.1:64015 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:30] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 511, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.06
[2026-10-16 13:59:31] INFO:     127.0.0.1:64024 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64025 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64026 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64027 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64028 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64029 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64030 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64031 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64032 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64033 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64034 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64035 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64036 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64037 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64038 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64039 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64040 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64041 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64042 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64043 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64044 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64045 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64046 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64047 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 511, token usage: 0.00, #running-req: 0, #queue-req: 20, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.83
[2026-10-16 13:59:31] INFO:     127.0.0.1:64048 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64049 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64050 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64051 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64052 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64053 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64054 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:31] INFO:     127.0.0.1:64055 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:32] Prefill batch, #new-seq: 20, #new-token: 20, #cached-token: 9916, token usage: 0.06, #running-req: 1, #queue-req: 11, #pending-token: 0, cuda graph: False, input throughput (token/s): 75.41
[2026-10-16 13:59:33] Prefill batch, #new-seq: 11, #new-token: 11, #cached-token: 5617, token usage: 0.09, #running-req: 21, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 8.47
[2026-10-16 13:59:33] Decode batch, #running-req: 32, #token: 16095, token usage: 0.09, cuda graph: False, gen throughput (token/s): 16.25, #queue-req: 0
[2026-10-16 13:59:36] Decode batch, #running-req: 32, #token: 16095, token usage: 0.09, cuda graph: False, gen throughput (token/s): 442.87, #queue-req: 0
[2026-10-16 13:59:39] Decode batch, #running-req: 32, #token: 16095, token usage: 0.09, cuda graph: False, gen throughput (token/s): 409.79, #queue-req: 0
[2026-10-16 13:59:43] Decode batch, #running-req: 32, #token: 0, token usage: 0.00, cuda graph: False, gen throughput (token/s): 407.60, #queue-req: 0
[2026-10-16 13:59:43] INFO:     127.0.0.1:64074 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64075 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64076 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64077 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64078 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64079 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64080 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64081 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64082 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64083 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64084 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64085 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64086 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64087 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64088 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64089 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64090 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64091 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64092 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64093 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64094 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64095 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64096 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64097 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64098 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64099 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64100 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64101 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64102 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64103 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64104 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] INFO:     127.0.0.1:64105 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 13:59:43] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 23, #pending-token: 0, cuda graph: False, input throughput (token/s): 50.84
[2026-10-16 13:59:45] Prefill batch, #new-seq: 9, #new-token: 4096, #cached-token: 0, token usage: 0.03, #running-req: 1, #queue-req: 22, #pending-token: 7886, cuda graph: False, input throughput (token/s): 2291.41
[2026-10-16 13:59:47] Prefill batch, #new-seq: 8, #new-token: 4096, #cached-token: 3, token usage: 0.05, #running-req: 9, #queue-req: 15, #pending-token: 7690, cuda graph: False, input throughput (token/s): 2166.56
[2026-10-16 13:59:48] Prefill batch, #new-seq: 9, #new-token: 4096, #cached-token: 5, token usage: 0.07, #running-req: 16, #queue-req: 7, #pending-token: 3589, cuda graph: False, input throughput (token/s): 2277.02
[2026-10-16 13:59:50] Prefill batch, #new-seq: 8, #new-token: 3584, #cached-token: 5, token usage: 0.09, #running-req: 24, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 1997.12
[2026-10-16 13:59:54] Decode batch, #running-req: 32, #token: 16425, token usage: 0.09, cuda graph: False, gen throughput (token/s): 114.23, #queue-req: 0
[2026-10-16 13:59:58] Decode batch, #running-req: 32, #token: 16425, token usage: 0.09, cuda graph: False, gen throughput (token/s): 307.91, #queue-req: 0
[2026-10-16 14:00:02] Decode batch, #running-req: 32, #token: 16425, token usage: 0.09, cuda graph: False, gen throughput (token/s): 302.20, #queue-req: 0
[2026-10-16 14:00:03] INFO:     127.0.0.1:64141 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64142 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64143 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64144 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64145 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64146 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64147 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64148 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64149 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64150 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64151 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64152 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64153 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64154 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64155 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64156 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64157 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64158 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64159 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64160 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64161 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64162 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64163 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64164 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64165 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64166 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64167 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64168 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64169 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64170 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64171 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:03] INFO:     127.0.0.1:64172 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:04] Prefill batch, #new-seq: 1, #new-token: 510, #cached-token: 2, token usage: 0.00, #running-req: 0, #queue-req: 31, #pending-token: 0, cuda graph: False, input throughput (token/s): 38.14
[2026-10-16 14:00:05] Prefill batch, #new-seq: 9, #new-token: 4096, #cached-token: 10, token usage: 0.03, #running-req: 1, #queue-req: 22, #pending-token: 11700, cuda graph: False, input throughput (token/s): 2119.46
[2026-10-16 14:00:07] Prefill batch, #new-seq: 9, #new-token: 4096, #cached-token: 6, token usage: 0.05, #running-req: 9, #queue-req: 14, #pending-token: 7598, cuda graph: False, input throughput (token/s): 2052.78
[2026-10-16 14:00:09] Prefill batch, #new-seq: 9, #new-token: 4096, #cached-token: 19, token usage: 0.07, #running-req: 17, #queue-req: 6, #pending-token: 3483, cuda graph: False, input throughput (token/s): 2604.26
[2026-10-16 14:00:10] Prefill batch, #new-seq: 7, #new-token: 3474, #cached-token: 9, token usage: 0.09, #running-req: 25, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 2770.56
[2026-10-16 14:00:12] Decode batch, #running-req: 32, #token: 16317, token usage: 0.09, cuda graph: False, gen throughput (token/s): 126.86, #queue-req: 0
[2026-10-16 14:00:15] Decode batch, #running-req: 32, #token: 16317, token usage: 0.09, cuda graph: False, gen throughput (token/s): 409.51, #queue-req: 0
[2026-10-16 14:00:18] Decode batch, #running-req: 32, #token: 16317, token usage: 0.09, cuda graph: False, gen throughput (token/s): 445.12, #queue-req: 0
[2026-10-16 14:00:19] INFO:     127.0.0.1:64206 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64207 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64208 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64209 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64210 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64211 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64212 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64213 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64214 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64215 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64216 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64217 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64218 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64219 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64220 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64221 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64222 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64223 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64224 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64225 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64226 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64227 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64228 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64229 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64230 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64231 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64232 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64233 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64234 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64235 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64236 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:19] INFO:     127.0.0.1:64237 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:20] Prefill batch, #new-seq: 1, #new-token: 477, #cached-token: 1, token usage: 0.00, #running-req: 0, #queue-req: 31, #pending-token: 0, cuda graph: False, input throughput (token/s): 49.22
[2026-10-16 14:00:22] Prefill batch, #new-seq: 9, #new-token: 4096, #cached-token: 14, token usage: 0.03, #running-req: 1, #queue-req: 22, #pending-token: 11751, cuda graph: False, input throughput (token/s): 2126.58
[2026-10-16 14:00:24] Prefill batch, #new-seq: 9, #new-token: 4096, #cached-token: 14, token usage: 0.05, #running-req: 9, #queue-req: 14, #pending-token: 7641, cuda graph: False, input throughput (token/s): 2150.73
[2026-10-16 14:00:26] Prefill batch, #new-seq: 9, #new-token: 4096, #cached-token: 13, token usage: 0.07, #running-req: 17, #queue-req: 6, #pending-token: 3532, cuda graph: False, input throughput (token/s): 2063.77
[2026-10-16 14:00:28] Prefill batch, #new-seq: 7, #new-token: 3521, #cached-token: 11, token usage: 0.09, #running-req: 25, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 1921.37
[2026-10-16 14:00:29] Decode batch, #running-req: 32, #token: 16345, token usage: 0.09, cuda graph: False, gen throughput (token/s): 118.88, #queue-req: 0
[2026-10-16 14:00:32] Decode batch, #running-req: 32, #token: 16345, token usage: 0.09, cuda graph: False, gen throughput (token/s): 495.86, #queue-req: 0
[2026-10-16 14:00:35] Decode batch, #running-req: 32, #token: 16345, token usage: 0.09, cuda graph: False, gen throughput (token/s): 446.42, #queue-req: 0
[2026-10-16 14:00:36] INFO:     127.0.0.1:64273 - "GET /server_info HTTP/1.1" 200 OK
[2026-10-16 14:00:36] INFO:     127.0.0.1:64274 - "GET /server_info HTTP/1.1" 200 OK
[2026-10-16 14:00:44] INFO:     127.0.0.1:64316 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-16 14:00:44] INFO:     127.0.0.1:64317 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-16 14:00:46] INFO:     127.0.0.1:64333 - "GET /model_info HTTP/1.1" 200 OK
[2026-10-16 14:00:48] INFO:     127.0.0.1:64338 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:51] Prefill batch, #new-seq: 1, #new-token: 2287, #cached-token: 0, token usage: 0.01, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 97.67
[2026-10-16 14:00:51] Decode batch, #running-req: 1, #token: 2288, token usage: 0.01, cuda graph: False, gen throughput (token/s): 47.13, #queue-req: 0
[2026-10-16 14:00:52] INFO:     127.0.0.1:64351 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:52] INFO:     127.0.0.1:64352 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:52] INFO:     127.0.0.1:64353 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:52] INFO:     127.0.0.1:64354 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:52] INFO:     127.0.0.1:64355 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:52] INFO:     127.0.0.1:64356 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:52] INFO:     127.0.0.1:64357 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:52] INFO:     127.0.0.1:64358 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:53] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 2286, token usage: 0.01, #running-req: 0, #queue-req: 7, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.64
[2026-10-16 14:00:54] Prefill batch, #new-seq: 2, #new-token: 4096, #cached-token: 0, token usage: 0.04, #running-req: 1, #queue-req: 5, #pending-token: 11979, cuda graph: False, input throughput (token/s): 4015.33
[2026-10-16 14:00:55] Prefill batch, #new-seq: 3, #new-token: 4096, #cached-token: 0, token usage: 0.05, #running-req: 2, #queue-req: 3, #pending-token: 7883, cuda graph: False, input throughput (token/s): 3810.88
[2026-10-16 14:00:55] Prefill batch, #new-seq: 4, #new-token: 1410, #cached-token: 6473, token usage: 0.05, #running-req: 4, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 1710.13
[2026-10-16 14:00:57] Decode batch, #running-req: 8, #token: 9730, token usage: 0.05, cuda graph: False, gen throughput (token/s): 37.04, #queue-req: 0
[2026-10-16 14:00:59] Decode batch, #running-req: 8, #token: 0, token usage: 0.00, cuda graph: False, gen throughput (token/s): 161.51, #queue-req: 0
[2026-10-16 14:00:59] INFO:     127.0.0.1:64379 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:59] INFO:     127.0.0.1:64380 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:59] INFO:     127.0.0.1:64381 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:59] INFO:     127.0.0.1:64382 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:59] INFO:     127.0.0.1:64383 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:59] INFO:     127.0.0.1:64384 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:59] INFO:     127.0.0.1:64385 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:59] INFO:     127.0.0.1:64386 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:00:59] Prefill batch, #new-seq: 1, #new-token: 134, #cached-token: 2153, token usage: 0.01, #running-req: 0, #queue-req: 7, #pending-token: 0, cuda graph: False, input throughput (token/s): 39.27
[2026-10-16 14:01:00] Prefill batch, #new-seq: 7, #new-token: 955, #cached-token: 15102, token usage: 0.05, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 1472.94
[2026-10-16 14:01:02] Decode batch, #running-req: 8, #token: 9724, token usage: 0.05, cuda graph: False, gen throughput (token/s): 98.39, #queue-req: 0
[2026-10-16 14:01:03] INFO:     127.0.0.1:64415 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:03] INFO:     127.0.0.1:64416 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:04] INFO:     127.0.0.1:64417 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:04] INFO:     127.0.0.1:64418 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:04] INFO:     127.0.0.1:64419 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:04] INFO:     127.0.0.1:64420 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:04] INFO:     127.0.0.1:64421 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:04] INFO:     127.0.0.1:64422 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:04] Prefill batch, #new-seq: 1, #new-token: 133, #cached-token: 2154, token usage: 0.01, #running-req: 0, #queue-req: 7, #pending-token: 0, cuda graph: False, input throughput (token/s): 32.60
[2026-10-16 14:01:04] Prefill batch, #new-seq: 7, #new-token: 938, #cached-token: 15101, token usage: 0.05, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 1443.61
[2026-10-16 14:01:06] Decode batch, #running-req: 8, #token: 9706, token usage: 0.05, cuda graph: False, gen throughput (token/s): 84.97, #queue-req: 0
[2026-10-16 14:01:09] Decode batch, #running-req: 8, #token: 9706, token usage: 0.05, cuda graph: False, gen throughput (token/s): 103.37, #queue-req: 0
[2026-10-16 14:01:09] INFO:     127.0.0.1:64439 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:09] INFO:     127.0.0.1:64440 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:10] INFO:     127.0.0.1:64441 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:10] INFO:     127.0.0.1:64442 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:10] INFO:     127.0.0.1:64443 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:10] INFO:     127.0.0.1:64444 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:10] INFO:     127.0.0.1:64445 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:10] INFO:     127.0.0.1:64446 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:10] Prefill batch, #new-seq: 1, #new-token: 132, #cached-token: 2167, token usage: 0.01, #running-req: 0, #queue-req: 7, #pending-token: 0, cuda graph: False, input throughput (token/s): 24.19
[2026-10-16 14:01:11] Prefill batch, #new-seq: 7, #new-token: 944, #cached-token: 15072, token usage: 0.05, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 900.50
[2026-10-16 14:01:13] Decode batch, #running-req: 8, #token: 9711, token usage: 0.05, cuda graph: False, gen throughput (token/s): 79.80, #queue-req: 0
[2026-10-16 14:01:15] INFO:     127.0.0.1:64465 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:15] INFO:     127.0.0.1:64466 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:15] INFO:     127.0.0.1:64467 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:15] INFO:     127.0.0.1:64468 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:15] INFO:     127.0.0.1:64469 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:15] INFO:     127.0.0.1:64470 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:15] INFO:     127.0.0.1:64471 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:15] INFO:     127.0.0.1:64472 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:16] Prefill batch, #new-seq: 1, #new-token: 142, #cached-token: 2167, token usage: 0.01, #running-req: 0, #queue-req: 7, #pending-token: 0, cuda graph: False, input throughput (token/s): 29.82
[2026-10-16 14:01:17] Prefill batch, #new-seq: 7, #new-token: 939, #cached-token: 15087, token usage: 0.05, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 850.73
[2026-10-16 14:01:17] Decode batch, #running-req: 8, #token: 9716, token usage: 0.05, cuda graph: False, gen throughput (token/s): 73.90, #queue-req: 0
[2026-10-16 14:01:20] Decode batch, #running-req: 8, #token: 9716, token usage: 0.05, cuda graph: False, gen throughput (token/s): 119.51, #queue-req: 0
[2026-10-16 14:01:21] INFO:     127.0.0.1:64495 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:21] INFO:     127.0.0.1:64496 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:21] INFO:     127.0.0.1:64497 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:21] INFO:     127.0.0.1:64498 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:21] INFO:     127.0.0.1:64499 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:21] INFO:     127.0.0.1:64500 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:21] INFO:     127.0.0.1:64501 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:21] INFO:     127.0.0.1:64502 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:21] Prefill batch, #new-seq: 1, #new-token: 138, #cached-token: 2167, token usage: 0.01, #running-req: 0, #queue-req: 7, #pending-token: 0, cuda graph: False, input throughput (token/s): 29.53
[2026-10-16 14:01:23] Prefill batch, #new-seq: 7, #new-token: 926, #cached-token: 15085, token usage: 0.04, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 770.70
[2026-10-16 14:01:24] Decode batch, #running-req: 8, #token: 7545, token usage: 0.04, cuda graph: False, gen throughput (token/s): 76.42, #queue-req: 0
[2026-10-16 14:01:28] Decode batch, #running-req: 8, #token: 0, token usage: 0.00, cuda graph: False, gen throughput (token/s): 88.08, #queue-req: 0
[2026-10-16 14:01:28] INFO:     127.0.0.1:64539 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:28] INFO:     127.0.0.1:64540 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:28] INFO:     127.0.0.1:64541 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:28] INFO:     127.0.0.1:64542 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:28] INFO:     127.0.0.1:64543 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:28] INFO:     127.0.0.1:64544 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:28] INFO:     127.0.0.1:64545 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:28] INFO:     127.0.0.1:64546 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:28] Prefill batch, #new-seq: 1, #new-token: 142, #cached-token: 2153, token usage: 0.01, #running-req: 0, #queue-req: 7, #pending-token: 0, cuda graph: False, input throughput (token/s): 25.67
[2026-10-16 14:01:29] Prefill batch, #new-seq: 7, #new-token: 930, #cached-token: 15103, token usage: 0.05, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 1005.17
[2026-10-16 14:01:32] Decode batch, #running-req: 8, #token: 9707, token usage: 0.05, cuda graph: False, gen throughput (token/s): 71.82, #queue-req: 0
[2026-10-16 14:01:34] INFO:     127.0.0.1:64576 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:34] INFO:     127.0.0.1:64577 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:34] INFO:     127.0.0.1:64578 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:34] INFO:     127.0.0.1:64579 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:34] INFO:     127.0.0.1:64580 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:34] INFO:     127.0.0.1:64581 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:34] INFO:     127.0.0.1:64582 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:34] INFO:     127.0.0.1:64583 - "POST /generate HTTP/1.1" 200 OK
[2026-10-16 14:01:35] Prefill batch, #new-seq: 2, #new-token: 267, #cached-token: 4307, token usage: 0.03, #running-req: 0, #queue-req: 6, #pending-token: 0, cuda graph: False, input throughput (token/s): 46.03
[2026-10-16 14:01:36] Prefill batch, #new-seq: 6, #new-token: 799, #cached-token: 12947, token usage: 0.05, #running-req: 2, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 965.34
[2026-10-16 14:01:37] Decode batch, #running-req: 8, #token: 9701, token usage: 0.05, cuda graph: False, gen throughput (token/s): 70.71, #queue-req: 0
[2026-10-16 14:01:40] Decode batch, #running-req: 8, #token: 9701, token usage: 0.05, cuda graph: False, gen throughput (token/s): 98.83, #queue-req: 0
[2026-10-16 14:01:41] INFO:     127.0.0.1:64605 - "GET /server_info HTTP/1.1" 200 OK
[2026-10-16 14:01:42] INFO:     127.0.0.1:64614 - "GET /server_info HTTP/1.1" 200 OK
[2026-10-16 14:01:45] INFO:     127.0.0.1:64685 - "GET /metrics HTTP/1.1" 200 OK
[2026-10-16 14:01:45] SIGTERM received. signum=None frame=None. Draining requests and shutting down...
[2026-10-16 14:01:45] Subprocess scheduler_0 (pid=5909) crashed with exit code -15. Triggering SIGQUIT for cleanup...
[2026-10-16 14:01:45] SIGQUIT received. signum=None, frame=None. It usually means one child failed.
[2026-10-16 14:01:46] Sleeping 5 seconds before crash diagnostics to let GPU activity settle.
[2026-10-16 14:01:51] No live scheduler processes found; skipping py-spy and CUDA coredump.
[2026-10-16 14:01:51] kill_process_tree called: parent_pid=5782, include_parent=True, pid=5782