[2026-10-17 15:34:30] Registered linear attn model: config=GigaChat35Config, backend=GDNAttnBackend, archs=['GigaChat35ForCausalLM', 'GigaChat35ForCausalLMNextN']
[2026-10-17 15:34:33] FlashInfer allreduce fusion is forcibly disabled via --enforce-disable-flashinfer-allreduce-fusion.
[2026-10-17 15:34:33] Sampling backend is set to pytorch for deterministic inference.
[2026-10-17 15:34:33] Attention backend not specified. Falling back to 'fa3' for deterministic inference. You can explicitly set --attention-backend to one of ['ascend', 'fa3', 'fa4', 'flashinfer', 'intel_xpu', 'triton'].
[2026-10-17 15:34:34] server_args={'model_path': 'Qwen/Qwen3-8B', 'tokenizer_path': 'Qwen/Qwen3-8B', 'tokenizer_mode': 'auto', 'tokenizer_backend': 'huggingface', 'tokenizer_worker_num': 1, 'detokenizer_worker_num': 1, 'skip_tokenizer_init': False, 'load_format': 'auto', 'model_loader_extra_config': '{}', 'trust_remote_code': False, 'context_length': None, 'is_embedding': False, 'enable_multimodal': None, 'revision': None, 'model_impl': 'auto', 'model_config_parser': 'auto', 'json_model_override_args': '{}', 'dtype': 'auto', 'quantization': None, 'quantization_param_path': None, 'kv_cache_dtype': 'auto', 'enable_fp32_lm_head': False, 'modelopt_quant': None, 'modelopt_checkpoint_restore_path': None, 'modelopt_checkpoint_save_path': None, 'modelopt_export_path': None, 'quantize_and_serve': False, 'rl_quant_profile': None, 'enable_tf32_matmul': False, 'mem_fraction_static': 0.85, 'max_running_requests': None, 'max_queued_requests': None, 'max_total_tokens': None, 'chunked_prefill_size': 4096, 'prefill_decode_interval': 0, 'enable_dynamic_chunking': False, 'max_prefill_tokens': 16384, 'prefill_max_requests': None, 'schedule_policy': 'fcfs', 'enable_priority_scheduling': False, 'disable_priority_preemption': False, 'default_priority_value': None, 'abort_on_priority_when_disabled': False, 'schedule_low_priority_values_first': False, 'priority_scheduling_preemption_threshold': 10, 'retraction_policy': 'length', 'schedule_conservativeness': 1.0, 'page_size': 1, 'c128_page_size': 16, 'swa_full_tokens_ratio': 0.8, 'disable_hybrid_swa_memory': False, 'radix_eviction_policy': 'lru', 'radix_eviction_policy_config': None, 'prefill_only_disable_kv_cache': False, 'disable_radix_cache': False, 'enable_page_major_kv_layout': False, 'enable_unified_memory': False, 'disable_chunked_prefix_cache': False, 'disable_overlap_schedule': False, 'num_continuous_decode_steps': 1, 'scheduler_recv_interval': 1, 'enable_mixed_chunk': False, 'nccl_port': None, 'dist_timeout': None, 'dist_init_addr': None, 'gated_launch_port': None, 'nnodes': 1, 'node_rank': 0, 'tp_size': 1, 'dcp_size': 1, 'pp_size': 1, 'pp_max_micro_batch_size': None, 'pp_async_batch_depth': 0, 'dp_size': 1, 'attn_dp_size': 1, 'load_balance_method': 'round_robin', 'attn_cp_size': 1, 'moe_dp_size': 1, 'dwdp_size': 1, 'dcp_comm_backend': 'ag_rs', 'dcp_replicate_q_proj': None, 'enable_prefill_cp': False, 'cp_strategy': None, 'enable_dsa_cache_layer_split': False, 'enable_cp_decode_attn_tp': False, 'enable_kv_cache_sharding': False, 'enable_dp_attention': False, 'enable_dp_attention_local_control_broadcast': False, 'enable_dp_lm_head': False, 'enable_tp_lm_head_all_to_all': False, 'enable_attn_tp_input_scattered': False, 'enable_shared_experts_attn_tp': False, 'shared_experts_tp_size': None, 'enable_dense_mlp_attn_tp': False, 'enable_layernorm_sp': False, 'disable_attn_tp_gather': False, 'enable_p2p_check': False, 'device': 'cuda', 'base_gpu_id': 0, 'gpu_id_step': 1, 'random_seed': 496161435, 'mlx_enable_sampling': False, 'watchdog_timeout': 300, 'soft_watchdog_timeout': None, 'sleep_on_idle': False, 'use_ray': False, 'custom_sigquit_handler': None, 'numa_node': None, 'gc_threshold': None, 'host': '127.0.0.1', 'port': 30000, 'fastapi_root_path': '', 'smg_grpc_mode': False, 'grpc_mode': False, 'grpc_port': None, 'grpc_worker_threads': 4, 'sidecar': None, 'sidecar_args': None, 'skip_server_warmup': False, 'warmups': None, 'enable_http2': False, 'http2_max_concurrent_streams': 200, 'http2_initial_connection_window_size': 1048576, 'ssl_keyfile': None, 'ssl_certfile': None, 'ssl_ca_certs': None, 'ssl_keyfile_password': None, 'enable_ssl_refresh': False, 'api_key': None, 'admin_api_key': None, 'served_model_name': 'Qwen/Qwen3-8B', 'weight_version': 'default', 'chat_template': None, 'trust_request_chat_template': False, 'hf_chat_template_name': None, 'completion_template': None, 'file_storage_path': None, 'enable_cache_report': True, 'return_input_ids': False, 'return_output_ids': False, 'reasoning_parser': None, 'default_chat_template_kwargs': None, 'strip_thinking_cache': False, 'enable_strict_thinking': False, 'tool_call_parser': None, 'tool_server': None, 'sampling_defaults': 'model', 'asr_max_buffer_seconds': 60, 'asr_max_concurrent_sessions': 32, 'preferred_sampling_params': None, 'allow_auto_truncate': False, 'stream_interval': 1, 'batch_notify_size': 16, 'stream_response_default_include_usage': False, 'incremental_streaming_output': False, 'enable_streaming_session': False, 'enable_session_radix_cache': False, 'log_level': 'info', 'log_level_http': None, 'log_requests': False, 'log_requests_level': 2, 'log_requests_format': 'text', 'log_requests_target': None, 'uvicorn_access_log_exclude_prefixes': [], 'crash_dump_folder': None, 'show_time_cost': False, 'enable_metrics': True, 'smg_http_sidecar_port': None, 'enable_mfu_metrics': False, 'enable_metrics_for_all_schedulers': False, 'load_snapshot_publish_interval': 15, 'tokenizer_metrics_custom_labels_header': 'x-custom-labels', 'tokenizer_metrics_allowed_custom_labels': None, 'extra_metric_labels': None, 'bucket_time_to_first_token': None, 'bucket_inter_token_latency': None, 'bucket_e2e_request_latency': None, 'prompt_tokens_buckets': None, 'generation_tokens_buckets': None, 'gc_warning_threshold_secs': 0.0, 'decode_log_interval': 40, 'enable_request_time_stats_logging': False, 'kv_events_config': None, 'load_publish_endpoint': None, 'enable_forward_pass_metrics': False, 'forward_pass_metrics_worker_id': '', 'forward_pass_metrics_ipc_name': None, 'enable_trace': False, 'trace_modules': 'request', 'otlp_traces_endpoint': 'localhost:4317', 'export_metrics_to_file': False, 'export_metrics_to_file_dir': None, 'stat_loggers': None, 'constrained_json_whitespace_pattern': None, 'constrained_json_disable_any_whitespace': False, 'constrained_json_max_whitespace_cnt': None, 'attention_backend': 'fa3', 'decode_attention_backend': None, 'enable_lean_attention': None, 'prefill_attention_backend': None, 'sampling_backend': 'pytorch', 'grammar_backend': 'xgrammar', 'radix_cache_backend': None, 'mm_attention_backend': None, 'fp8_gemm_runner_backend': 'auto', 'fp4_gemm_runner_backend': 'auto', 'bf16_gemm_backend': 'auto', 'dsa_prefill_backend': None, 'dsv4_prefill_backend': 'auto', 'dsa_decode_backend': None, 'dsa_paged_mqa_logits_backend': 'auto', 'dsa_topk_backend': 'sgl-kernel', 'disable_flashinfer_autotune': False, 'flashinfer_autotune_skip_ops': None, 'mamba_backend': 'triton', 'cuda_graph_config': {'decode': {'backend': 'full', 'max_bs': 48, 'bs': [1, 2, 4, 8, 12, 16, 24, 32, 40, 48], 'tc_compiler': 'eager', 'max_context_size': None, 'full_prefill_max_req': None, 'full_prefill_prefix_chunk_tokens': None, 'max_seq_len': None}, 'prefill': {'backend': 'breakable', 'max_bs': 4096, 'bs': [4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048, 2304, 2560, 2816, 3072, 3328, 3584, 3840, 4096], 'tc_compiler': 'eager', 'max_context_size': None, 'full_prefill_max_req': None, 'full_prefill_prefix_chunk_tokens': None, 'max_seq_len': None}}, 'cuda_graph_backend_decode': None, 'cuda_graph_backend_prefill': None, 'cuda_graph_max_bs_decode': None, 'cuda_graph_max_bs_prefill': None, 'cuda_graph_bs_decode': None, 'cuda_graph_bs_prefill': None, 'cuda_graph_prefill_max_context': None, 'disable_prefill_cuda_graph': False, 'disable_decode_cuda_graph': False, 'disable_cuda_graph': False, 'disable_cuda_graph_padding': False, 'enable_profile_cuda_graph': False, 'enable_cudagraph_gc': False, 'debug_cuda_graph': False, 'enable_layerwise_nvtx_marker': False, 'enable_nccl_nvls': False, 'enable_symm_mem': False, 'triton_attention_reduce_in_fp32': False, 'triton_attention_num_kv_splits': 8, 'triton_attention_split_tile_size': None, 'flashinfer_mla_disable_ragged': False, 'enable_fused_qk_norm_rope': False, 'enable_precise_embedding_interpolation': False, 'enable_fused_moe_sum_all_reduce': False, 'enable_deepseek_v4_fp4_indexer': False, 'disable_custom_all_reduce': False, 'enable_mscclpp': False, 'enable_torch_symm_mem': False, 'enable_scattered_sconv': False, 'pre_warm_nccl': False, 'boundary_reduction': 'ar', 'enable_quant_communications': False, 'enable_flashinfer_allreduce_fusion': False, 'enforce_disable_flashinfer_allreduce_fusion': True, 'flashinfer_allreduce_fusion_backend': None, 'enable_aiter_allreduce_fusion': False, 'enable_torch_compile': False, 'enable_torch_compile_debug_mode': False, 'torch_compile_max_bs': 32, 'speculative_algorithm': None, 'uno_lora_path': None, 'speculative_draft_model_path': None, 'speculative_draft_model_revision': None, 'speculative_draft_load_format': None, 'speculative_num_steps': None, 'speculative_eagle_topk': None, 'speculative_num_draft_tokens': None, 'speculative_dflash_block_size': None, 'speculative_dspark_block_size': None, 'speculative_dspark_sps_table_path': None, 'speculative_dspark_confidence_sts_path': None, 'speculative_dspark_align_verify_tokens_to_graph_tier': False, 'speculative_accept_threshold_single': 1.0, 'speculative_accept_threshold_acc': 1.0, 'speculative_use_rejection_sampling': False, 'speculative_token_map': None, 'speculative_attention_mode': 'prefill', 'speculative_draft_attention_backend': None, 'speculative_dsa_topk_backend': 'sgl-kernel', 'speculative_draft_kv_cache_dtype': None, 'speculative_draft_window_size': None, 'speculative_moe_runner_backend': 'auto', 'speculative_moe_a2a_backend': None, 'speculative_enable_w4a4_mxfp4_megamoe': None, 'speculative_draft_model_quantization': None, '_speculative_draft_quantization_explicitly_set': False, 'speculative_skip_dp_mlp_sync': False, 'enable_multi_layer_eagle': False, 'speculative_adaptive': False, 'speculative_adaptive_config': None, 'decoupled_spec_bind_endpoint': None, 'decoupled_spec_connect_endpoints': None, 'decoupled_spec_rank': None, 'decoupled_spec_role': 'null', 'spec_trace_dir': None, 'speculative_ngram_min_bfs_breadth': 1, 'speculative_ngram_max_bfs_breadth': 10, 'speculative_ngram_match_type': 'BFS', 'speculative_ngram_max_trie_depth': 18, 'speculative_ngram_capacity': 10000000, 'speculative_ngram_external_corpus_path': None, 'speculative_ngram_external_sam_budget': 0, 'speculative_ngram_external_corpus_max_tokens': 10000000, 'ep_size': 1, 'moe_a2a_backend': 'none', 'enable_w4a4_mxfp4_megamoe': False, 'deepep_v2_mode': 'direct', 'moe_runner_backend': 'auto', 'flashinfer_mxfp4_moe_precision': 'default', 'deepep_mode': 'auto', 'fuseep_mode': 2, 'deepep_dispatcher_output_dtype': 'auto', 'ep_num_redundant_experts': 0, 'ep_dispatch_algorithm': None, 'init_expert_location': 'trivial', 'enable_eplb': False, 'eplb_algorithm': 'auto', 'eplb_rebalance_num_iterations': 1000, 'eplb_rebalance_layers_per_chunk': None, 'eplb_min_rebalancing_utilization_threshold': 1.0, 'expert_distribution_recorder_mode': None, 'expert_distribution_recorder_buffer_size': 1000, 'expert_balancedness_report_mode': 'off', 'deepep_config': None, 'moe_dense_tp_size': None, 'elastic_ep_backend': None, 'enable_elastic_expert_backup': False, 'mooncake_ib_device': None, 'enable_waterfill': False, 'ep_join_mode': None, 'ep_join_rank_offset': 0, 'elastic_ep_initial_size': None, 'max_ep_size': None, 'elastic_ep_scale_timeout': 600, 'disable_flashinfer_cutlass_moe_fp4_allgather': False, 'disable_shared_experts_fusion': False, 'enforce_shared_experts_fusion': False, 'max_mamba_cache_size': None, 'mamba_ssm_dtype': None, 'mamba_max_states_per_path': -1, 'enable_mamba_cache_stochastic_rounding': False, 'mamba_cache_philox_rounds': 0, 'mamba_full_memory_ratio': 0.9, 'mamba_radix_cache_strategy': 'auto', 'uses_mamba_radix_cache': False, 'mamba_track_interval': 256, 'enable_int8_mamba_checkpoint': False, 'int8_mamba_ckpt_size': None, 'linear_attn_backend': 'triton', 'linear_attn_decode_backend': None, 'linear_attn_prefill_backend': None, 'linear_attn_verify_backend': None, 'enable_linear_replayssm': False, 'linear_replayssm_cache_len': 16, 'enable_linear_replayssm_spec': False, 'enable_hierarchical_cache': False, 'hicache_host_memory_mode': 'cache', 'hicache_ratio': 2.0, 'hicache_size': 0, 'hicache_write_policy': 'write_through', 'hicache_io_backend': 'kernel', 'hicache_mem_layout': 'page_first', 'hicache_storage_backend': None, 'hicache_storage_prefetch_policy': 'timeout', 'hicache_storage_backend_extra_config': None, 'hicache_storage_prefetch_retry_poll_interval': 8, 'hicache_storage_prefetch_retry_max_attempts': 8, 'enable_unified_cache_external_linker': False, 'unified_cache_external_linker_backend': 'mooncake', 'enable_hisparse': False, 'hisparse_config': None, 'enable_broadcast_mm_inputs_process': False, 'enable_prefix_mm_cache': False, 'mm_enable_dp_encoder': False, 'mm_process_config': {}, 'mm_processor_worker_num': 0, 'mm_io_worker_num': 0, 'allowed_media_domains': [], 'media_url_max_file_size_mb': 64, 'mm_preprocess_cache_size_mb': None, 'trust_mm_content_hashes': False, 'limit_mm_data_per_request': None, 'enable_mm_global_cache': False, 'image_processor_backend': 'auto', 'mm_global_cache_backend': 'mooncake', 'disable_fast_image_processor': False, 'mm_feature_transport': 'cpu', 'enable_lora': None, 'enable_lora_overlap_loading': None, 'max_lora_rank': None, 'lora_target_modules': None, 'lora_paths': None, 'max_loaded_loras': None, 'max_loras_per_batch': 8, 'lora_eviction_policy': 'lru', 'lora_backend': 'csgmv', 'max_lora_chunk_size': 16, 'experts_shared_outer_loras': None, 'lora_use_virtual_experts': False, 'lora_strict_loading': False, 'lora_drain_wait_threshold': 0.0, 'enable_two_batch_overlap': False, 'enable_single_batch_overlap': False, 'tbo_token_distribution_threshold': 0.48, 'cpu_offload_gb': 0, 'offload_group_size': -1, 'offload_num_in_group': 1, 'offload_prefetch_step': 1, 'offload_mode': 'cpu', 'enable_lmcache': False, 'lmcache_config_file': None, 'enable_flexkv': False, 'flexkv_config_file': None, 'kt_weight_path': None, 'kt_method': 'AMXINT4', 'kt_cpuinfer': None, 'kt_threadpool_count': 2, 'kt_num_gpu_experts': None, 'kt_max_deferred_experts_per_token': None, 'dllm_algorithm': None, 'dllm_algorithm_config': None, 'dllm_fdfo': True, 'disaggregation_mode': 'null', 'disaggregation_transfer_backend': 'mooncake', 'disaggregation_bootstrap_port': 8998, 'disaggregation_ib_device': None, 'disaggregation_decode_enable_radix_cache': False, 'disaggregation_decode_enable_offload_kvcache': False, 'disaggregation_decode_retraction_backup': None, 'num_reserved_decode_tokens': 512, 'disaggregation_decode_extra_slots': None, 'disaggregation_decode_polling_interval': 1, 'optimistic_prefill_attempts': 0, 'encoder_only': False, 'language_only': False, 'language_model_only': False, 'encoder_transfer_backend': 'zmq_to_scheduler', 'encoder_urls': [], 'encoder_bootstrap_port': 8997, 'encoder_register_urls': [], 'enable_adaptive_dispatch_to_encoder': False, 'enable_pdmux': False, 'pdmux_config_path': None, 'sm_group_num': 8, 'startup_weight_load_mode': 'serial', 'custom_weight_loader': [], 'weight_loader_disable_mmap': False, 'weight_loader_prefetch_checkpoints': False, 'weight_loader_prefetch_num_threads': 4, 'weight_loader_drop_cache_after_load': False, 'remote_instance_weight_loader_seed_instance_ip': None, 'remote_instance_weight_loader_seed_instance_service_port': None, 'remote_instance_weight_loader_send_weights_group_ports': None, 'remote_instance_weight_loader_backend': 'nccl', 'remote_instance_weight_loader_start_seed_via_transfer_engine': False, 'engine_info_bootstrap_port': 6789, 'modelexpress_config': None, 'download_dir': None, 'model_checksum': None, 'delete_ckpt_after_loading': False, 'decrypted_config_file': None, 'decrypted_draft_config_file': None, 'checkpoint_engine_wait_weights_before_ready': False, 'enable_prefill_delayer': False, 'prefill_delayer_max_delay_passes': 30, 'prefill_delayer_token_usage_low_watermark': None, 'prefill_delayer_forward_passes_buckets': None, 'prefill_delayer_wait_seconds_buckets': None, 'prefill_delayer_queue_min_ratio': None, 'prefill_delayer_max_delay_ms': None, 'min_free_slots_delay': None, 'enable_deterministic_inference': True, 'rl_on_policy_target': None, 'kv_canary': 'none', 'kv_canary_real_data': 'none', 'kv_canary_sweep_interval': 0, 'enable_dynamic_batch_tokenizer': False, 'dynamic_batch_tokenizer_batch_size': 32, 'dynamic_batch_tokenizer_batch_timeout': 0.002, 'enable_tokenizer_batch_encode': False, 'disable_tokenizer_batch_decode': False, 'debug_tensor_dump_output_folder': None, 'debug_tensor_dump_layers': None, 'debug_tensor_dump_input_file': None, 'enable_memory_saver': False, 'enable_weights_cpu_backup': False, 'enable_draft_weights_cpu_backup': False, 'enable_custom_logit_processor': False, 'enable_return_hidden_states': False, 'return_hidden_states_mode': None, 'enable_return_routed_experts': False, 'enable_return_indexer_topk': False, 'sampling_mask_max_tokens': 4096, 'disable_outlines_disk_cache': False, 'enable_mis': False, 'weight_cache_mode': 'off', 'weight_cache_socket': None, 'weight_cache_timeout': 1800, 'forward_hooks': None, 'msprobe_dump_config': None, 'ple_offload_embedding': None, 'ple_offload_backend': 'pinned', 'ple_offload_dir': None, 'disaggregation_enable_kv_checksum': False, 'enable_pd_role_switch': False, 'speculative_domino_candidate_pool_size': 2048, 'flashinfer_a2a_dispatch_type': None, 'cuda_graph_max_seq_len_prefill': None, 'prefill_kv_cache_dequant_dtype': 'auto', 'dsv4_attn_backend': 'auto', 'otlp_service_name': None, '_radix_eviction_policy_explicitly_set': False, 'hicache_host_memory_fraction': 0.8, 'enable_linker_mla_dedup': False, '_swa_full_tokens_ratio_explicitly_set': False, 'swa_prefix_tails': None, 'enable_encoder_swa_bounded_replay': False, 'enable_decoder_swa_bounded_replay': False, 'enable_response_store': False, 'disaggregation_decode_host_receive_threshold': 1.0, 'qsa_indexer_dtype': 'auto', 'disaggregation_decode_allocation_policy': 'early', 'speculative_boundary_reduction': 'ar', 'speculative_use_block_verification': False, 'speculative_draft_sink_size': None, 'disable_aiter_allreduce_fusion_in_prefill': False, 'disable_aiter_allreduce_fusion_in_decode': False, 'dsa_triton_union': 0, 'enable_dsa_fused_indexer': None, 'enable_cp_tp_group_sharing': False, 'enable_watermark': False, 'watermark_config': '<redacted>', 'watermark_key': '<redacted>', 'watermark_key_b': '<redacted>', 'watermark_context_window': 4, 'watermark_mixing_probability': 0.5, 'watermark_max_probability': 1.0, 'watermark_default_enabled': False, 'watermark_enforce_all': False, 'grpc_response_timeout_secs': 300}
[2026-10-17 15:34:35] Using default HuggingFace chat template with detected content format: string
[2026-10-17 15:34:36] Auto-detected template features: reasoning_config=ReasoningToggleConfig(toggle_param='enable_thinking', default_enabled=True, special_case=None, effort_kwarg=None), reasoning_parser=qwen3, tool_call_parser=qwen
[2026-10-17 15:34:44] Init parallel begin.
[2026-10-17 15:34:44] Init parallel ends. elapsed=0.21 s
[2026-10-17 15:34:45] Memory baseline taken. mem usage=0.00 GB
/data/sglang/.venv/lib/python3.12/site-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`.
  warnings.warn(
/data/sglang/.venv/lib/python3.12/site-packages/torch/jit/_script.py:1643: FutureWarning: `torch.jit.interface` is deprecated. Please use `torch.compile` instead.
  warnings.warn(
/data/sglang/.venv/lib/python3.12/site-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`.
  warnings.warn(
[2026-10-17 15:34:46] Load weight begin. avail mem=21.69 GB
[2026-10-17 15:34:46] Found local HF snapshot for Qwen/Qwen3-8B at /data/sglang/hf/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218; skipping download.

Multi-thread loading shards:   0% Completed | 0/5 [00:00<?, ?it/s]
Multi-thread loading shards:  20% Completed | 1/5 [00:00<00:02,  1.35it/s]
Multi-thread loading shards:  40% Completed | 2/5 [00:01<00:02,  1.20it/s]
Multi-thread loading shards:  60% Completed | 3/5 [00:02<00:01,  1.23it/s]
Multi-thread loading shards:  80% Completed | 4/5 [00:03<00:00,  1.32it/s]
Multi-thread loading shards: 100% Completed | 5/5 [00:03<00:00,  1.65it/s]
Multi-thread loading shards: 100% Completed | 5/5 [00:03<00:00,  1.46it/s]
[2026-10-17 15:34:50] Load weight end. elapsed=3.63 s, type=Qwen3ForCausalLM, avail mem=6.41 GB, mem usage=15.28 GB.
[2026-10-17 15:34:51] KV Cache is allocated. dtype: torch.bfloat16, #tokens: 22939, K size: 1.58 GB, V size: 1.58 GB
[2026-10-17 15:34:51] Memory pool end. avail mem=2.86 GB
[2026-10-17 15:34:52] Capture target prefill CUDA graph begin. backend=breakable, num_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048, 2304, 2560, 2816, 3072, 3328, 3584, 3840, 4096], avail mem=2.86 GB

  0%|          | 0/50 [00:00<?, ?it/s]
Capturing prefill shape (num_tokens=4096 avail_mem=2.86 GB):   0%|          | 0/50 [00:00<?, ?it/s]
Capturing prefill shape (num_tokens=4096 avail_mem=2.86 GB):   2%|▏         | 1/50 [00:03<02:49,  3.45s/it]
Capturing prefill shape (num_tokens=3840 avail_mem=1.33 GB):   2%|▏         | 1/50 [00:03<02:49,  3.45s/it]
Capturing prefill shape (num_tokens=3840 avail_mem=1.33 GB):   4%|▍         | 2/50 [00:06<02:45,  3.45s/it]
Capturing prefill shape (num_tokens=3584 avail_mem=1.32 GB):   4%|▍         | 2/50 [00:06<02:45,  3.45s/it]
Capturing prefill shape (num_tokens=3584 avail_mem=1.32 GB):   6%|▌         | 3/50 [00:10<02:45,  3.53s/it]
Capturing prefill shape (num_tokens=3328 avail_mem=1.30 GB):   6%|▌         | 3/50 [00:10<02:45,  3.53s/it]
Capturing prefill shape (num_tokens=3328 avail_mem=1.30 GB):   8%|▊         | 4/50 [00:13<02:40,  3.48s/it]
Capturing prefill shape (num_tokens=3072 avail_mem=1.29 GB):   8%|▊         | 4/50 [00:13<02:40,  3.48s/it]
Capturing prefill shape (num_tokens=3072 avail_mem=1.29 GB):  10%|█         | 5/50 [00:17<02:38,  3.53s/it]
Capturing prefill shape (num_tokens=2816 avail_mem=1.28 GB):  10%|█         | 5/50 [00:17<02:38,  3.53s/it]
Capturing prefill shape (num_tokens=2816 avail_mem=1.28 GB):  12%|█▏        | 6/50 [00:20<02:30,  3.43s/it]
Capturing prefill shape (num_tokens=2560 avail_mem=1.26 GB):  12%|█▏        | 6/50 [00:20<02:30,  3.43s/it]
Capturing prefill shape (num_tokens=2560 avail_mem=1.26 GB):  14%|█▍        | 7/50 [00:23<02:20,  3.26s/it]
Capturing prefill shape (num_tokens=2304 avail_mem=1.24 GB):  14%|█▍        | 7/50 [00:23<02:20,  3.26s/it]
Capturing prefill shape (num_tokens=2304 avail_mem=1.24 GB):  16%|█▌        | 8/50 [00:26<02:16,  3.24s/it]
Capturing prefill shape (num_tokens=2048 avail_mem=1.23 GB):  16%|█▌        | 8/50 [00:26<02:16,  3.24s/it]
Capturing prefill shape (num_tokens=2048 avail_mem=1.23 GB):  18%|█▊        | 9/50 [00:29<02:04,  3.03s/it]
Capturing prefill shape (num_tokens=1792 avail_mem=1.22 GB):  18%|█▊        | 9/50 [00:29<02:04,  3.03s/it]
Capturing prefill shape (num_tokens=1792 avail_mem=1.22 GB):  20%|██        | 10/50 [00:31<01:49,  2.74s/it]
Capturing prefill shape (num_tokens=1536 avail_mem=1.20 GB):  20%|██        | 10/50 [00:31<01:49,  2.74s/it]
Capturing prefill shape (num_tokens=1536 avail_mem=1.20 GB):  22%|██▏       | 11/50 [00:33<01:35,  2.45s/it]
Capturing prefill shape (num_tokens=1280 avail_mem=1.19 GB):  22%|██▏       | 11/50 [00:33<01:35,  2.45s/it]
Capturing prefill shape (num_tokens=1280 avail_mem=1.19 GB):  24%|██▍       | 12/50 [00:35<01:25,  2.26s/it]
Capturing prefill shape (num_tokens=1024 avail_mem=1.17 GB):  24%|██▍       | 12/50 [00:35<01:25,  2.26s/it]
Capturing prefill shape (num_tokens=1024 avail_mem=1.17 GB):  26%|██▌       | 13/50 [00:36<01:11,  1.94s/it]
Capturing prefill shape (num_tokens=960 avail_mem=1.16 GB):  26%|██▌       | 13/50 [00:36<01:11,  1.94s/it] 
Capturing prefill shape (num_tokens=960 avail_mem=1.16 GB):  28%|██▊       | 14/50 [00:37<01:02,  1.74s/it]
Capturing prefill shape (num_tokens=896 avail_mem=1.14 GB):  28%|██▊       | 14/50 [00:37<01:02,  1.74s/it]
Capturing prefill shape (num_tokens=896 avail_mem=1.14 GB):  30%|███       | 15/50 [00:38<00:52,  1.51s/it]
Capturing prefill shape (num_tokens=832 avail_mem=1.13 GB):  30%|███       | 15/50 [00:38<00:52,  1.51s/it]
Capturing prefill shape (num_tokens=832 avail_mem=1.13 GB):  32%|███▏      | 16/50 [00:39<00:47,  1.39s/it]
Capturing prefill shape (num_tokens=768 avail_mem=1.11 GB):  32%|███▏      | 16/50 [00:39<00:47,  1.39s/it]
Capturing prefill shape (num_tokens=768 avail_mem=1.11 GB):  34%|███▍      | 17/50 [00:40<00:44,  1.34s/it]
Capturing prefill shape (num_tokens=704 avail_mem=1.10 GB):  34%|███▍      | 17/50 [00:40<00:44,  1.34s/it]
Capturing prefill shape (num_tokens=704 avail_mem=1.10 GB):  36%|███▌      | 18/50 [00:41<00:39,  1.25s/it]
Capturing prefill shape (num_tokens=640 avail_mem=1.08 GB):  36%|███▌      | 18/50 [00:41<00:39,  1.25s/it]
Capturing prefill shape (num_tokens=640 avail_mem=1.08 GB):  38%|███▊      | 19/50 [00:42<00:33,  1.10s/it]
Capturing prefill shape (num_tokens=576 avail_mem=1.07 GB):  38%|███▊      | 19/50 [00:42<00:33,  1.10s/it]
Capturing prefill shape (num_tokens=576 avail_mem=1.07 GB):  40%|████      | 20/50 [00:43<00:29,  1.03it/s]
Capturing prefill shape (num_tokens=512 avail_mem=1.05 GB):  40%|████      | 20/50 [00:43<00:29,  1.03it/s]
Capturing prefill shape (num_tokens=512 avail_mem=1.05 GB):  42%|████▏     | 21/50 [00:44<00:25,  1.13it/s]
Capturing prefill shape (num_tokens=480 avail_mem=1.03 GB):  42%|████▏     | 21/50 [00:44<00:25,  1.13it/s]
Capturing prefill shape (num_tokens=480 avail_mem=1.03 GB):  44%|████▍     | 22/50 [00:44<00:22,  1.25it/s]
Capturing prefill shape (num_tokens=448 avail_mem=1.02 GB):  44%|████▍     | 22/50 [00:44<00:22,  1.25it/s]
Capturing prefill shape (num_tokens=448 avail_mem=1.02 GB):  46%|████▌     | 23/50 [00:45<00:20,  1.32it/s]
Capturing prefill shape (num_tokens=416 avail_mem=1.01 GB):  46%|████▌     | 23/50 [00:45<00:20,  1.32it/s]
Capturing prefill shape (num_tokens=416 avail_mem=1.01 GB):  48%|████▊     | 24/50 [00:45<00:18,  1.39it/s]
Capturing prefill shape (num_tokens=384 avail_mem=0.99 GB):  48%|████▊     | 24/50 [00:45<00:18,  1.39it/s]
Capturing prefill shape (num_tokens=384 avail_mem=0.99 GB):  50%|█████     | 25/50 [00:46<00:16,  1.54it/s]
Capturing prefill shape (num_tokens=352 avail_mem=0.98 GB):  50%|█████     | 25/50 [00:46<00:16,  1.54it/s]
Capturing prefill shape (num_tokens=352 avail_mem=0.98 GB):  52%|█████▏    | 26/50 [00:46<00:14,  1.66it/s]
Capturing prefill shape (num_tokens=320 avail_mem=0.96 GB):  52%|█████▏    | 26/50 [00:46<00:14,  1.66it/s]
Capturing prefill shape (num_tokens=320 avail_mem=0.96 GB):  54%|█████▍    | 27/50 [00:47<00:13,  1.75it/s]
Capturing prefill shape (num_tokens=288 avail_mem=0.95 GB):  54%|█████▍    | 27/50 [00:47<00:13,  1.75it/s]
Capturing prefill shape (num_tokens=288 avail_mem=0.95 GB):  56%|█████▌    | 28/50 [00:47<00:11,  1.95it/s]
Capturing prefill shape (num_tokens=256 avail_mem=0.93 GB):  56%|█████▌    | 28/50 [00:47<00:11,  1.95it/s]
Capturing prefill shape (num_tokens=256 avail_mem=0.93 GB):  58%|█████▊    | 29/50 [00:48<00:08,  2.34it/s]
Capturing prefill shape (num_tokens=240 avail_mem=0.92 GB):  58%|█████▊    | 29/50 [00:48<00:08,  2.34it/s]
Capturing prefill shape (num_tokens=240 avail_mem=0.92 GB):  60%|██████    | 30/50 [00:48<00:07,  2.62it/s]
Capturing prefill shape (num_tokens=224 avail_mem=0.90 GB):  60%|██████    | 30/50 [00:48<00:07,  2.62it/s]
Capturing prefill shape (num_tokens=224 avail_mem=0.90 GB):  62%|██████▏   | 31/50 [00:48<00:07,  2.63it/s]
Capturing prefill shape (num_tokens=208 avail_mem=0.89 GB):  62%|██████▏   | 31/50 [00:48<00:07,  2.63it/s]
Capturing prefill shape (num_tokens=208 avail_mem=0.89 GB):  64%|██████▍   | 32/50 [00:49<00:07,  2.54it/s]
Capturing prefill shape (num_tokens=192 avail_mem=0.87 GB):  64%|██████▍   | 32/50 [00:49<00:07,  2.54it/s]
Capturing prefill shape (num_tokens=192 avail_mem=0.87 GB):  66%|██████▌   | 33/50 [00:49<00:06,  2.48it/s]
Capturing prefill shape (num_tokens=176 avail_mem=0.86 GB):  66%|██████▌   | 33/50 [00:49<00:06,  2.48it/s]
Capturing prefill shape (num_tokens=176 avail_mem=0.86 GB):  68%|██████▊   | 34/50 [00:49<00:06,  2.45it/s]
Capturing prefill shape (num_tokens=160 avail_mem=0.85 GB):  68%|██████▊   | 34/50 [00:49<00:06,  2.45it/s]
Capturing prefill shape (num_tokens=160 avail_mem=0.85 GB):  70%|███████   | 35/50 [00:50<00:06,  2.41it/s]
Capturing prefill shape (num_tokens=144 avail_mem=0.83 GB):  70%|███████   | 35/50 [00:50<00:06,  2.41it/s]
Capturing prefill shape (num_tokens=144 avail_mem=0.83 GB):  72%|███████▏  | 36/50 [00:50<00:05,  2.50it/s]
Capturing prefill shape (num_tokens=128 avail_mem=0.81 GB):  72%|███████▏  | 36/50 [00:50<00:05,  2.50it/s]
Capturing prefill shape (num_tokens=128 avail_mem=0.81 GB):  74%|███████▍  | 37/50 [00:51<00:04,  2.89it/s]
Capturing prefill shape (num_tokens=112 avail_mem=0.79 GB):  74%|███████▍  | 37/50 [00:51<00:04,  2.89it/s]
Capturing prefill shape (num_tokens=112 avail_mem=0.79 GB):  76%|███████▌  | 38/50 [00:51<00:03,  3.24it/s]
Capturing prefill shape (num_tokens=96 avail_mem=0.77 GB):  76%|███████▌  | 38/50 [00:51<00:03,  3.24it/s] 
Capturing prefill shape (num_tokens=96 avail_mem=0.77 GB):  78%|███████▊  | 39/50 [00:51<00:03,  3.51it/s]
Capturing prefill shape (num_tokens=80 avail_mem=0.76 GB):  78%|███████▊  | 39/50 [00:51<00:03,  3.51it/s]
Capturing prefill shape (num_tokens=80 avail_mem=0.76 GB):  80%|████████  | 40/50 [00:51<00:02,  3.73it/s]
Capturing prefill shape (num_tokens=64 avail_mem=0.74 GB):  80%|████████  | 40/50 [00:51<00:02,  3.73it/s]
Capturing prefill shape (num_tokens=64 avail_mem=0.74 GB):  82%|████████▏ | 41/50 [00:51<00:02,  3.89it/s]
Capturing prefill shape (num_tokens=48 avail_mem=0.73 GB):  82%|████████▏ | 41/50 [00:51<00:02,  3.89it/s]
Capturing prefill shape (num_tokens=48 avail_mem=0.73 GB):  84%|████████▍ | 42/50 [00:52<00:02,  3.97it/s]
Capturing prefill shape (num_tokens=32 avail_mem=0.71 GB):  84%|████████▍ | 42/50 [00:52<00:02,  3.97it/s]
Capturing prefill shape (num_tokens=32 avail_mem=0.71 GB):  86%|████████▌ | 43/50 [00:52<00:01,  4.27it/s]
Capturing prefill shape (num_tokens=28 avail_mem=0.70 GB):  86%|████████▌ | 43/50 [00:52<00:01,  4.27it/s]
Capturing prefill shape (num_tokens=28 avail_mem=0.70 GB):  88%|████████▊ | 44/50 [00:52<00:01,  4.51it/s]
Capturing prefill shape (num_tokens=24 avail_mem=0.68 GB):  88%|████████▊ | 44/50 [00:52<00:01,  4.51it/s]
Capturing prefill shape (num_tokens=24 avail_mem=0.68 GB):  90%|█████████ | 45/50 [00:52<00:01,  4.71it/s]
Capturing prefill shape (num_tokens=20 avail_mem=0.67 GB):  90%|█████████ | 45/50 [00:52<00:01,  4.71it/s]
Capturing prefill shape (num_tokens=20 avail_mem=0.67 GB):  92%|█████████▏| 46/50 [00:52<00:00,  4.85it/s]
Capturing prefill shape (num_tokens=16 avail_mem=0.65 GB):  92%|█████████▏| 46/50 [00:52<00:00,  4.85it/s]
Capturing prefill shape (num_tokens=16 avail_mem=0.65 GB):  94%|█████████▍| 47/50 [00:53<00:00,  4.54it/s]
Capturing prefill shape (num_tokens=12 avail_mem=0.64 GB):  94%|█████████▍| 47/50 [00:53<00:00,  4.54it/s]
Capturing prefill shape (num_tokens=12 avail_mem=0.64 GB):  96%|█████████▌| 48/50 [00:53<00:00,  4.68it/s]
Capturing prefill shape (num_tokens=8 avail_mem=0.63 GB):  96%|█████████▌| 48/50 [00:53<00:00,  4.68it/s] 
Capturing prefill shape (num_tokens=8 avail_mem=0.63 GB):  98%|█████████▊| 49/50 [00:53<00:00,  4.69it/s]
Capturing prefill shape (num_tokens=4 avail_mem=0.61 GB):  98%|█████████▊| 49/50 [00:53<00:00,  4.69it/s]
Capturing prefill shape (num_tokens=4 avail_mem=0.61 GB): 100%|██████████| 50/50 [00:53<00:00,  4.68it/s]
Capturing prefill shape (num_tokens=4 avail_mem=0.61 GB): 100%|██████████| 50/50 [00:53<00:00,  1.08s/it]
[2026-10-17 15:35:46] Capture target prefill CUDA graph end. elapsed=54.54 s, mem usage=1.50 GB, avail mem=1.36 GB.
[2026-10-17 15:35:46] Capture target decode CUDA graph begin. backend=full, num_tokens_per_req=1, avail mem=1.36 GB

  0%|          | 0/10 [00:00<?, ?it/s]
Capturing batches (bs=48 avail_mem=1.34 GB):   0%|          | 0/10 [00:00<?, ?it/s]
Capturing batches (bs=48 avail_mem=1.34 GB):  10%|█         | 1/10 [00:00<00:02,  4.11it/s]
Capturing batches (bs=40 avail_mem=1.33 GB):  10%|█         | 1/10 [00:00<00:02,  4.11it/s]
Capturing batches (bs=40 avail_mem=1.33 GB):  20%|██        | 2/10 [00:00<00:01,  4.35it/s]
Capturing batches (bs=32 avail_mem=1.32 GB):  20%|██        | 2/10 [00:00<00:01,  4.35it/s]
Capturing batches (bs=32 avail_mem=1.32 GB):  30%|███       | 3/10 [00:00<00:01,  4.45it/s]
Capturing batches (bs=24 avail_mem=1.31 GB):  30%|███       | 3/10 [00:00<00:01,  4.45it/s]
Capturing batches (bs=24 avail_mem=1.31 GB):  40%|████      | 4/10 [00:00<00:01,  4.51it/s]
Capturing batches (bs=16 avail_mem=1.30 GB):  40%|████      | 4/10 [00:00<00:01,  4.51it/s]
Capturing batches (bs=16 avail_mem=1.30 GB):  50%|█████     | 5/10 [00:01<00:01,  4.46it/s]
Capturing batches (bs=12 avail_mem=1.29 GB):  50%|█████     | 5/10 [00:01<00:01,  4.46it/s]
Capturing batches (bs=12 avail_mem=1.29 GB):  60%|██████    | 6/10 [00:01<00:00,  4.22it/s]
Capturing batches (bs=8 avail_mem=1.28 GB):  60%|██████    | 6/10 [00:01<00:00,  4.22it/s] 
Capturing batches (bs=8 avail_mem=1.28 GB):  70%|███████   | 7/10 [00:01<00:00,  4.03it/s]
Capturing batches (bs=4 avail_mem=1.27 GB):  70%|███████   | 7/10 [00:01<00:00,  4.03it/s]
Capturing batches (bs=4 avail_mem=1.27 GB):  80%|████████  | 8/10 [00:01<00:00,  3.91it/s]
Capturing batches (bs=2 avail_mem=1.26 GB):  80%|████████  | 8/10 [00:01<00:00,  3.91it/s]
Capturing batches (bs=2 avail_mem=1.26 GB):  90%|█████████ | 9/10 [00:02<00:00,  3.89it/s]
Capturing batches (bs=1 avail_mem=1.25 GB):  90%|█████████ | 9/10 [00:02<00:00,  3.89it/s]
Capturing batches (bs=1 avail_mem=1.25 GB): 100%|██████████| 10/10 [00:02<00:00,  3.85it/s]
Capturing batches (bs=1 avail_mem=1.25 GB): 100%|██████████| 10/10 [00:02<00:00,  4.07it/s]
[2026-10-17 15:35:49] Capture target decode CUDA graph end. bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48], elapsed=3.20 s, mem usage=0.12 GB, avail mem=1.24 GB.
[2026-10-17 15:35:49] max_total_num_tokens=22939, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=2048, context_len=40960, available_gpu_mem=1.24 GB
[2026-10-17 15:35:49] Init Unified Radix Cache. Components: (<ComponentType.FULL: 0>,). Tree Core: RustUnifiedTreeCore
[2026-10-17 15:35:49] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=False hybrid_ssm=False hicache_attached=False
[2026-10-17 15:35:50] Engine startup timings (s): load_weight=3.63, kv_cache_allocation=0.38, scheduler_e2e=65.78, cuda_graph={prefill=54.54, decode=3.20, target_verify=0.00, draft_prefill=0.00, draft_decode=0.00, draft_extend=0.00}, tokenizer_e2e=75.42
[2026-10-17 15:35:50] INFO:     Started server process [1353539]
[2026-10-17 15:35:50] INFO:     Waiting for application startup.
[2026-10-17 15:35:50] Using default chat sampling params from model generation config: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
[2026-10-17 15:35:50] INFO:     Application startup complete.
[2026-10-17 15:35:50] INFO:     Uvicorn running on http://127.0.0.1:30000 (Press CTRL+C to quit)
[2026-10-17 15:35:50] INFO:     127.0.0.1:58636 - "GET /health HTTP/1.1" 503 Service Unavailable
[2026-10-17 15:35:51] INFO:     127.0.0.1:58652 - "GET /model_info HTTP/1.1" 200 OK
[2026-10-17 15:35:51] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 3.41
[2026-10-17 15:35:52] INFO:     127.0.0.1:58656 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:35:52] INFO:     127.0.0.1:58662 - "POST /freeze_gc HTTP/1.1" 200 OK
[2026-10-17 15:35:52] The server is fired up and ready to roll!
[2026-10-17 15:35:52] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.74
[2026-10-17 15:35:53] INFO:     127.0.0.1:58664 - "GET /health HTTP/1.1" 200 OK
[2026-10-17 15:35:53] Prefill batch, #new-seq: 1, #new-token: 24, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 22.54
[2026-10-17 15:35:56] Decode batch, #running-req: 1, #token: 58, token usage: 0.00, cuda graph: True, gen throughput (token/s): 5.67, #queue-req: 0
[2026-10-17 15:36:01] Decode batch, #running-req: 1, #token: 98, token usage: 0.00, cuda graph: True, gen throughput (token/s): 7.82, #queue-req: 0
[2026-10-17 15:36:07] Decode batch, #running-req: 1, #token: 138, token usage: 0.01, cuda graph: True, gen throughput (token/s): 7.70, #queue-req: 0
[2026-10-17 15:36:08] INFO:     127.0.0.1:58678 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:36:09] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 23, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.07
[2026-10-17 15:36:12] Decode batch, #running-req: 1, #token: 51, token usage: 0.00, cuda graph: True, gen throughput (token/s): 7.58, #queue-req: 0
[2026-10-17 15:36:17] Decode batch, #running-req: 1, #token: 91, token usage: 0.00, cuda graph: True, gen throughput (token/s): 7.76, #queue-req: 0
[2026-10-17 15:36:22] Decode batch, #running-req: 1, #token: 131, token usage: 0.01, cuda graph: True, gen throughput (token/s): 7.64, #queue-req: 0
[2026-10-17 15:36:25] INFO:     127.0.0.1:53254 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:36:25] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 23, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.06
[2026-10-17 15:36:28] Decode batch, #running-req: 1, #token: 44, token usage: 0.00, cuda graph: True, gen throughput (token/s): 7.55, #queue-req: 0
[2026-10-17 15:36:33] Decode batch, #running-req: 1, #token: 84, token usage: 0.00, cuda graph: True, gen throughput (token/s): 7.69, #queue-req: 0
[2026-10-17 15:36:38] Decode batch, #running-req: 1, #token: 124, token usage: 0.01, cuda graph: True, gen throughput (token/s): 7.66, #queue-req: 0
[2026-10-17 15:36:42] INFO:     127.0.0.1:57424 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:36:42] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 23, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.06
[2026-10-17 15:36:43] Decode batch, #running-req: 1, #token: 37, token usage: 0.00, cuda graph: True, gen throughput (token/s): 7.50, #queue-req: 0
[2026-10-17 15:36:49] Decode batch, #running-req: 1, #token: 77, token usage: 0.00, cuda graph: True, gen throughput (token/s): 7.60, #queue-req: 0
[2026-10-17 15:36:54] Decode batch, #running-req: 1, #token: 117, token usage: 0.01, cuda graph: True, gen throughput (token/s): 7.64, #queue-req: 0
[2026-10-17 15:36:58] INFO:     127.0.0.1:40404 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:36:59] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 23, token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.06
[2026-10-17 15:36:59] Decode batch, #running-req: 1, #token: 30, token usage: 0.00, cuda graph: True, gen throughput (token/s): 7.36, #queue-req: 0
[2026-10-17 15:37:05] Decode batch, #running-req: 1, #token: 70, token usage: 0.00, cuda graph: True, gen throughput (token/s): 7.62, #queue-req: 0
[2026-10-17 15:37:10] Decode batch, #running-req: 1, #token: 110, token usage: 0.00, cuda graph: True, gen throughput (token/s): 7.70, #queue-req: 0
[2026-10-17 15:37:16] Decode batch, #running-req: 1, #token: 150, token usage: 0.01, cuda graph: True, gen throughput (token/s): 6.72, #queue-req: 0
[2026-10-17 15:37:16] INFO:     127.0.0.1:43704 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:37:16] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 23, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.06
[2026-10-17 15:37:17] Prefill batch, #new-seq: 15, #new-token: 341, #cached-token: 0, token usage: 0.02, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 733.37
[2026-10-17 15:37:23] Decode batch, #running-req: 16, #token: 954, token usage: 0.04, cuda graph: True, gen throughput (token/s): 81.42, #queue-req: 0
[2026-10-17 15:37:29] Decode batch, #running-req: 16, #token: 1594, token usage: 0.07, cuda graph: True, gen throughput (token/s): 114.81, #queue-req: 0
[2026-10-17 15:37:34] Decode batch, #running-req: 16, #token: 2234, token usage: 0.10, cuda graph: True, gen throughput (token/s): 112.65, #queue-req: 0
[2026-10-17 15:37:36] INFO:     127.0.0.1:47502 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:37:40] Decode batch, #running-req: 15, #token: 2691, token usage: 0.12, cuda graph: True, gen throughput (token/s): 105.95, #queue-req: 0
[2026-10-17 15:37:46] Decode batch, #running-req: 15, #token: 3291, token usage: 0.14, cuda graph: True, gen throughput (token/s): 103.72, #queue-req: 0
[2026-10-17 15:37:52] Decode batch, #running-req: 15, #token: 3891, token usage: 0.17, cuda graph: True, gen throughput (token/s): 103.03, #queue-req: 0
[2026-10-17 15:37:54] INFO:     127.0.0.1:47514 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:37:54] INFO:     127.0.0.1:47530 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:37:54] INFO:     127.0.0.1:47534 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:37:54] INFO:     127.0.0.1:47538 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:37:54] INFO:     127.0.0.1:47544 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:37:54] INFO:     127.0.0.1:47556 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:37:54] INFO:     127.0.0.1:47560 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:37:54] INFO:     127.0.0.1:47562 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:37:54] INFO:     127.0.0.1:47570 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:37:54] INFO:     127.0.0.1:47576 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:37:54] INFO:     127.0.0.1:47588 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:37:54] INFO:     127.0.0.1:47598 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:37:54] INFO:     127.0.0.1:47606 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:37:54] INFO:     127.0.0.1:47616 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:37:54] INFO:     127.0.0.1:47618 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:37:54] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 23, token usage: 0.01, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.03
[2026-10-17 15:37:55] Prefill batch, #new-seq: 15, #new-token: 15, #cached-token: 326, token usage: 0.02, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 106.01
[2026-10-17 15:37:58] Decode batch, #running-req: 16, #token: 714, token usage: 0.03, cuda graph: True, gen throughput (token/s): 103.33, #queue-req: 0
[2026-10-17 15:38:04] Decode batch, #running-req: 16, #token: 1354, token usage: 0.06, cuda graph: True, gen throughput (token/s): 112.36, #queue-req: 0
[2026-10-17 15:38:09] Decode batch, #running-req: 16, #token: 1994, token usage: 0.09, cuda graph: True, gen throughput (token/s): 112.74, #queue-req: 0
[2026-10-17 15:38:13] INFO:     127.0.0.1:39796 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:38:15] Decode batch, #running-req: 15, #token: 2466, token usage: 0.11, cuda graph: True, gen throughput (token/s): 108.03, #queue-req: 0
[2026-10-17 15:38:21] Decode batch, #running-req: 15, #token: 3066, token usage: 0.13, cuda graph: True, gen throughput (token/s): 103.43, #queue-req: 0
[2026-10-17 15:38:27] Decode batch, #running-req: 15, #token: 3666, token usage: 0.16, cuda graph: True, gen throughput (token/s): 103.02, #queue-req: 0
[2026-10-17 15:38:31] INFO:     127.0.0.1:39812 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:38:31] INFO:     127.0.0.1:39814 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:38:31] INFO:     127.0.0.1:39830 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:38:31] INFO:     127.0.0.1:39842 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:38:31] INFO:     127.0.0.1:39836 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:38:31] INFO:     127.0.0.1:39850 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:38:31] INFO:     127.0.0.1:39858 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:38:31] INFO:     127.0.0.1:39856 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:38:31] INFO:     127.0.0.1:39862 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:38:31] INFO:     127.0.0.1:39866 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:38:31] INFO:     127.0.0.1:39876 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:38:31] INFO:     127.0.0.1:39878 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:38:31] INFO:     127.0.0.1:39882 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:38:31] INFO:     127.0.0.1:39892 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:38:31] INFO:     127.0.0.1:39904 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:38:31] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 23, token usage: 0.01, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.03
[2026-10-17 15:38:32] Prefill batch, #new-seq: 15, #new-token: 15, #cached-token: 326, token usage: 0.02, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 106.40
[2026-10-17 15:38:33] Decode batch, #running-req: 16, #token: 474, token usage: 0.02, cuda graph: True, gen throughput (token/s): 100.32, #queue-req: 0
[2026-10-17 15:38:38] Decode batch, #running-req: 16, #token: 1114, token usage: 0.05, cuda graph: True, gen throughput (token/s): 112.16, #queue-req: 0
[2026-10-17 15:38:44] Decode batch, #running-req: 16, #token: 1754, token usage: 0.08, cuda graph: True, gen throughput (token/s): 112.33, #queue-req: 0
[2026-10-17 15:38:50] INFO:     127.0.0.1:35426 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:38:50] Decode batch, #running-req: 15, #token: 2241, token usage: 0.10, cuda graph: True, gen throughput (token/s): 110.58, #queue-req: 0
[2026-10-17 15:38:56] Decode batch, #running-req: 15, #token: 2841, token usage: 0.12, cuda graph: True, gen throughput (token/s): 103.31, #queue-req: 0
[2026-10-17 15:39:01] Decode batch, #running-req: 15, #token: 3441, token usage: 0.15, cuda graph: True, gen throughput (token/s): 103.41, #queue-req: 0
[2026-10-17 15:39:07] Decode batch, #running-req: 15, #token: 4041, token usage: 0.18, cuda graph: True, gen throughput (token/s): 102.45, #queue-req: 0
[2026-10-17 15:39:08] INFO:     127.0.0.1:35434 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:08] INFO:     127.0.0.1:35446 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:08] INFO:     127.0.0.1:35454 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:08] INFO:     127.0.0.1:35460 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:08] INFO:     127.0.0.1:35470 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:08] INFO:     127.0.0.1:35482 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:08] INFO:     127.0.0.1:35484 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:08] INFO:     127.0.0.1:35492 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:08] INFO:     127.0.0.1:35508 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:08] INFO:     127.0.0.1:35522 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:08] INFO:     127.0.0.1:35532 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:08] INFO:     127.0.0.1:35540 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:08] INFO:     127.0.0.1:35556 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:08] INFO:     127.0.0.1:35568 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:08] INFO:     127.0.0.1:35580 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:08] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 23, token usage: 0.01, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.03
[2026-10-17 15:39:09] Prefill batch, #new-seq: 15, #new-token: 15, #cached-token: 326, token usage: 0.02, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 106.11
[2026-10-17 15:39:13] Decode batch, #running-req: 16, #token: 874, token usage: 0.04, cuda graph: True, gen throughput (token/s): 104.82, #queue-req: 0
[2026-10-17 15:39:19] Decode batch, #running-req: 16, #token: 1514, token usage: 0.07, cuda graph: True, gen throughput (token/s): 111.24, #queue-req: 0
[2026-10-17 15:39:25] Decode batch, #running-req: 16, #token: 2154, token usage: 0.09, cuda graph: True, gen throughput (token/s): 111.07, #queue-req: 0
[2026-10-17 15:39:27] INFO:     127.0.0.1:60496 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:31] Decode batch, #running-req: 15, #token: 2616, token usage: 0.11, cuda graph: True, gen throughput (token/s): 105.50, #queue-req: 0
[2026-10-17 15:39:36] Decode batch, #running-req: 15, #token: 3216, token usage: 0.14, cuda graph: True, gen throughput (token/s): 103.27, #queue-req: 0
[2026-10-17 15:39:42] Decode batch, #running-req: 15, #token: 3816, token usage: 0.17, cuda graph: True, gen throughput (token/s): 102.99, #queue-req: 0
[2026-10-17 15:39:46] INFO:     127.0.0.1:60502 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:46] INFO:     127.0.0.1:60508 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:46] INFO:     127.0.0.1:60524 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:46] INFO:     127.0.0.1:60534 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:46] INFO:     127.0.0.1:60544 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:46] INFO:     127.0.0.1:60552 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:46] INFO:     127.0.0.1:60556 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:46] INFO:     127.0.0.1:60572 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:46] INFO:     127.0.0.1:60566 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:46] INFO:     127.0.0.1:60576 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:46] INFO:     127.0.0.1:60582 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:46] INFO:     127.0.0.1:60586 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:46] INFO:     127.0.0.1:60590 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:46] INFO:     127.0.0.1:60600 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:46] INFO:     127.0.0.1:60608 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:39:46] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 23, token usage: 0.01, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.03
[2026-10-17 15:39:46] Prefill batch, #new-seq: 15, #new-token: 15, #cached-token: 326, token usage: 0.02, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 107.66
[2026-10-17 15:39:48] Decode batch, #running-req: 16, #token: 634, token usage: 0.03, cuda graph: True, gen throughput (token/s): 101.78, #queue-req: 0
[2026-10-17 15:39:54] Decode batch, #running-req: 16, #token: 1274, token usage: 0.06, cuda graph: True, gen throughput (token/s): 111.84, #queue-req: 0
[2026-10-17 15:40:01] Decode batch, #running-req: 16, #token: 1914, token usage: 0.08, cuda graph: True, gen throughput (token/s): 98.80, #queue-req: 0
[2026-10-17 15:40:05] INFO:     127.0.0.1:58744 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:06] Decode batch, #running-req: 15, #token: 2391, token usage: 0.10, cuda graph: True, gen throughput (token/s): 107.83, #queue-req: 0
[2026-10-17 15:40:12] Decode batch, #running-req: 15, #token: 2991, token usage: 0.13, cuda graph: True, gen throughput (token/s): 102.39, #queue-req: 0
[2026-10-17 15:40:18] Decode batch, #running-req: 15, #token: 3591, token usage: 0.16, cuda graph: True, gen throughput (token/s): 101.96, #queue-req: 0
[2026-10-17 15:40:24] INFO:     127.0.0.1:58746 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:24] INFO:     127.0.0.1:58754 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:24] INFO:     127.0.0.1:58758 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:24] INFO:     127.0.0.1:58764 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:24] INFO:     127.0.0.1:58776 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:24] INFO:     127.0.0.1:58782 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:24] INFO:     127.0.0.1:58788 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:24] INFO:     127.0.0.1:58794 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:24] INFO:     127.0.0.1:58798 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:24] INFO:     127.0.0.1:58804 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:24] INFO:     127.0.0.1:58812 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:24] INFO:     127.0.0.1:58826 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:24] INFO:     127.0.0.1:58840 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:24] INFO:     127.0.0.1:58852 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:24] INFO:     127.0.0.1:58860 - "POST /generate HTTP/1.1" 200 OK
/data/sglang/.venv/lib/python3.12/site-packages/sglang/srt/entrypoints/http_server.py:476: FastAPIDeprecationWarning: ORJSONResponse is deprecated, FastAPI now serializes data directly to JSON bytes via Pydantic when a return type or response model is set, which is faster and doesn't need a custom response class. Read more in the FastAPI docs: https://fastapi.tiangolo.com/advanced/custom-response/#orjson-or-response-model and https://fastapi.tiangolo.com/tutorial/response-model/
  return await original_handler(ORJSONRequest(request.scope, request.receive))
[2026-10-17 15:40:33] INFO:     127.0.0.1:58764 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-17 15:40:33] INFO:     127.0.0.1:58774 - "GET /v1/models HTTP/1.1" 200 OK
[2026-10-17 15:40:33] INFO:     127.0.0.1:58778 - "GET /model_info HTTP/1.1" 200 OK
[2026-10-17 15:40:41] INFO:     127.0.0.1:52516 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:42] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.02, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 9.18
[2026-10-17 15:40:42] Decode batch, #running-req: 1, #token: 516, token usage: 0.02, cuda graph: True, gen throughput (token/s): 23.54, #queue-req: 0
[2026-10-17 15:40:45] INFO:     127.0.0.1:39692 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:45] INFO:     127.0.0.1:39700 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:45] INFO:     127.0.0.1:39704 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:45] INFO:     127.0.0.1:39706 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:45] INFO:     127.0.0.1:39708 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:45] INFO:     127.0.0.1:39716 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:45] INFO:     127.0.0.1:39718 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:45] INFO:     127.0.0.1:39728 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:40:45] Prefill batch, #new-seq: 1, #new-token: 1, #cached-token: 511, token usage: 0.17, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 0.27
[2026-10-17 15:40:47] Prefill batch, #new-seq: 7, #new-token: 3327, #cached-token: 1, token usage: 0.17, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1756.95
[2026-10-17 15:40:48] Decode batch, #running-req: 8, #token: 3943, token usage: 0.17, cuda graph: True, gen throughput (token/s): 19.48, #queue-req: 0
[2026-10-17 15:40:52] Decode batch, #running-req: 8, #token: 4263, token usage: 0.19, cuda graph: True, gen throughput (token/s): 83.63, #queue-req: 0
[2026-10-17 15:40:56] Decode batch, #running-req: 8, #token: 4583, token usage: 0.20, cuda graph: True, gen throughput (token/s): 72.72, #queue-req: 0
[2026-10-17 15:41:02] INFO:     127.0.0.1:45944 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:02] INFO:     127.0.0.1:45950 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:02] INFO:     127.0.0.1:45954 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:02] INFO:     127.0.0.1:45968 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:02] INFO:     127.0.0.1:45982 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:02] INFO:     127.0.0.1:45988 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:02] INFO:     127.0.0.1:46000 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:02] INFO:     127.0.0.1:46006 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:02] Prefill batch, #new-seq: 1, #new-token: 496, #cached-token: 1, token usage: 0.07, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 33.35
[2026-10-17 15:41:03] Prefill batch, #new-seq: 2, #new-token: 1023, #cached-token: 0, token usage: 0.18, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1022.88
[2026-10-17 15:41:05] Prefill batch, #new-seq: 5, #new-token: 2556, #cached-token: 1, token usage: 0.18, #running-req: 3, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1120.74
[2026-10-17 15:41:06] Decode batch, #running-req: 8, #token: 4124, token usage: 0.18, cuda graph: True, gen throughput (token/s): 34.21, #queue-req: 0
[2026-10-17 15:41:10] Decode batch, #running-req: 8, #token: 4444, token usage: 0.19, cuda graph: True, gen throughput (token/s): 68.32, #queue-req: 0
[2026-10-17 15:41:15] Decode batch, #running-req: 8, #token: 4764, token usage: 0.21, cuda graph: True, gen throughput (token/s): 64.65, #queue-req: 0
[2026-10-17 15:41:20] Decode batch, #running-req: 8, #token: 5084, token usage: 0.22, cuda graph: True, gen throughput (token/s): 64.65, #queue-req: 0
[2026-10-17 15:41:21] INFO:     127.0.0.1:35402 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:21] INFO:     127.0.0.1:35412 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:21] INFO:     127.0.0.1:35428 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:21] INFO:     127.0.0.1:35432 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:21] INFO:     127.0.0.1:35436 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:21] INFO:     127.0.0.1:35450 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:21] INFO:     127.0.0.1:35462 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:21] INFO:     127.0.0.1:35464 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:21] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 1, token usage: 0.04, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 32.41
[2026-10-17 15:41:22] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.18, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1171.21
[2026-10-17 15:41:24] Prefill batch, #new-seq: 6, #new-token: 3041, #cached-token: 2, token usage: 0.18, #running-req: 2, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1251.69
[2026-10-17 15:41:30] Decode batch, #running-req: 8, #token: 4379, token usage: 0.19, cuda graph: True, gen throughput (token/s): 32.48, #queue-req: 0
[2026-10-17 15:41:36] Decode batch, #running-req: 8, #token: 4699, token usage: 0.20, cuda graph: True, gen throughput (token/s): 53.11, #queue-req: 0
[2026-10-17 15:41:41] Decode batch, #running-req: 8, #token: 5019, token usage: 0.22, cuda graph: True, gen throughput (token/s): 62.06, #queue-req: 0
[2026-10-17 15:41:43] INFO:     127.0.0.1:42986 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:43] INFO:     127.0.0.1:42990 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:43] INFO:     127.0.0.1:43000 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:43] INFO:     127.0.0.1:43010 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:43] INFO:     127.0.0.1:43026 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:43] INFO:     127.0.0.1:43032 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:43] INFO:     127.0.0.1:43038 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:43] INFO:     127.0.0.1:43046 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:41:43] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.07, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 26.45
[2026-10-17 15:41:44] Prefill batch, #new-seq: 2, #new-token: 1020, #cached-token: 3, token usage: 0.18, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 990.82
[2026-10-17 15:41:47] Prefill batch, #new-seq: 5, #new-token: 2552, #cached-token: 5, token usage: 0.18, #running-req: 3, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 983.40
[2026-10-17 15:41:52] Decode batch, #running-req: 8, #token: 4348, token usage: 0.19, cuda graph: True, gen throughput (token/s): 31.64, #queue-req: 0
[2026-10-17 15:41:58] Decode batch, #running-req: 8, #token: 4668, token usage: 0.20, cuda graph: True, gen throughput (token/s): 53.57, #queue-req: 0
[2026-10-17 15:42:03] Decode batch, #running-req: 8, #token: 4988, token usage: 0.22, cuda graph: True, gen throughput (token/s): 53.51, #queue-req: 0
[2026-10-17 15:42:06] INFO:     127.0.0.1:43716 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:06] INFO:     127.0.0.1:43732 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:06] INFO:     127.0.0.1:43744 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:06] INFO:     127.0.0.1:43750 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:06] INFO:     127.0.0.1:43762 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:06] INFO:     127.0.0.1:43774 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:06] INFO:     127.0.0.1:43778 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:06] INFO:     127.0.0.1:43792 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:06] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.07, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 26.32
[2026-10-17 15:42:07] Prefill batch, #new-seq: 2, #new-token: 1020, #cached-token: 0, token usage: 0.18, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1453.05
[2026-10-17 15:42:09] Prefill batch, #new-seq: 5, #new-token: 2559, #cached-token: 0, token usage: 0.18, #running-req: 3, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1438.72
[2026-10-17 15:42:12] Decode batch, #running-req: 8, #token: 4290, token usage: 0.19, cuda graph: True, gen throughput (token/s): 38.85, #queue-req: 0
[2026-10-17 15:42:18] Decode batch, #running-req: 8, #token: 4610, token usage: 0.20, cuda graph: True, gen throughput (token/s): 53.68, #queue-req: 0
[2026-10-17 15:42:24] Decode batch, #running-req: 8, #token: 4930, token usage: 0.21, cuda graph: True, gen throughput (token/s): 53.68, #queue-req: 0
[2026-10-17 15:42:27] INFO:     127.0.0.1:34784 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:27] INFO:     127.0.0.1:34800 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:27] INFO:     127.0.0.1:34810 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:27] INFO:     127.0.0.1:34818 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:27] INFO:     127.0.0.1:34834 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:27] INFO:     127.0.0.1:34840 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:27] INFO:     127.0.0.1:34848 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:27] INFO:     127.0.0.1:34858 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:28] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.04, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 27.34
[2026-10-17 15:42:28] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.19, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1032.88
[2026-10-17 15:42:31] Prefill batch, #new-seq: 6, #new-token: 3286, #cached-token: 2, token usage: 0.19, #running-req: 2, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1165.73
[2026-10-17 15:42:33] Decode batch, #running-req: 8, #token: 4456, token usage: 0.19, cuda graph: True, gen throughput (token/s): 35.17, #queue-req: 0
[2026-10-17 15:42:38] Decode batch, #running-req: 8, #token: 4776, token usage: 0.21, cuda graph: True, gen throughput (token/s): 66.94, #queue-req: 0
[2026-10-17 15:42:42] Decode batch, #running-req: 8, #token: 5096, token usage: 0.22, cuda graph: True, gen throughput (token/s): 64.47, #queue-req: 0
[2026-10-17 15:42:48] INFO:     127.0.0.1:45586 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:48] INFO:     127.0.0.1:45598 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:48] INFO:     127.0.0.1:45610 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:48] INFO:     127.0.0.1:45624 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:48] INFO:     127.0.0.1:45636 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:48] INFO:     127.0.0.1:45650 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:48] INFO:     127.0.0.1:45656 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:48] INFO:     127.0.0.1:45660 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:42:48] Prefill batch, #new-seq: 1, #new-token: 512, #cached-token: 0, token usage: 0.07, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 30.01
[2026-10-17 15:42:49] Prefill batch, #new-seq: 2, #new-token: 1024, #cached-token: 0, token usage: 0.18, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1376.75
[2026-10-17 15:42:51] Prefill batch, #new-seq: 5, #new-token: 2552, #cached-token: 3, token usage: 0.18, #running-req: 3, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1322.31
[2026-10-17 15:42:52] Decode batch, #running-req: 8, #token: 4178, token usage: 0.18, cuda graph: True, gen throughput (token/s): 34.72, #queue-req: 0
[2026-10-17 15:42:58] Decode batch, #running-req: 8, #token: 4498, token usage: 0.20, cuda graph: True, gen throughput (token/s): 49.15, #queue-req: 0
[2026-10-17 15:43:05] Decode batch, #running-req: 8, #token: 4818, token usage: 0.21, cuda graph: True, gen throughput (token/s): 48.25, #queue-req: 0
[2026-10-17 15:43:09] INFO:     127.0.0.1:33298 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:43:09] INFO:     127.0.0.1:33314 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:43:09] INFO:     127.0.0.1:33320 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:43:09] INFO:     127.0.0.1:33334 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:43:09] INFO:     127.0.0.1:33348 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:43:09] INFO:     127.0.0.1:33358 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:43:09] INFO:     127.0.0.1:33364 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:43:09] INFO:     127.0.0.1:33372 - "POST /generate HTTP/1.1" 200 OK
[2026-10-17 15:43:09] Prefill batch, #new-seq: 1, #new-token: 511, #cached-token: 0, token usage: 0.07, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 27.21
[2026-10-17 15:43:11] Prefill batch, #new-seq: 2, #new-token: 1005, #cached-token: 0, token usage: 0.17, #running-req: 1, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 886.22
[2026-10-17 15:43:13] Prefill batch, #new-seq: 5, #new-token: 2387, #cached-token: 0, token usage: 0.17, #running-req: 3, #queue-req: 0, #pending-token: 0, cuda graph: True, input throughput (token/s): 1056.67
[2026-10-17 15:43:13] Decode batch, #running-req: 8, #token: 3935, token usage: 0.17, cuda graph: True, gen throughput (token/s): 38.63, #queue-req: 0
[2026-10-17 15:43:19] Decode batch, #running-req: 8, #token: 4255, token usage: 0.19, cuda graph: True, gen throughput (token/s): 59.30, #queue-req: 0
[2026-10-17 15:43:24] Decode batch, #running-req: 8, #token: 4575, token usage: 0.20, cuda graph: True, gen throughput (token/s): 60.34, #queue-req: 0
[2026-10-17 15:43:29] Decode batch, #running-req: 8, #token: 4895, token usage: 0.21, cuda graph: True, gen throughput (token/s): 62.69, #queue-req: 0
[2026-10-17 15:43:29] INFO:     127.0.0.1:38986 - "GET /server_info HTTP/1.1" 200 OK
[2026-10-17 15:43:30] INFO:     127.0.0.1:38990 - "GET /server_info HTTP/1.1" 200 OK
[2026-10-17 15:43:32] SIGTERM received. signum=None frame=None. Draining requests and shutting down...
[2026-10-17 15:43:33] Subprocess scheduler_0 (pid=1353990) crashed with exit code -15. Triggering SIGQUIT for cleanup...
[2026-10-17 15:43:33] SIGQUIT received. signum=None, frame=None. It usually means one child failed.
[2026-10-17 15:43:33] Sleeping 5 seconds before crash diagnostics to let GPU activity settle.
[2026-10-17 15:43:38] No live scheduler processes found; skipping py-spy and CUDA coredump.
[2026-10-17 15:43:38] kill_process_tree called: parent_pid=1353539, include_parent=True, pid=1353539