功能详解Feature Tour

一览At a glance

Slurm / PBS 集群Slurm & PBS clusters

• 你的作业：用 squeue -u $USER 读取，显示名称、作业号、已运行时间 / 时间上限、节点和分区。 Your jobs, read with squeue -u $USER: name, ID, elapsed / time limit, nodes and partition.
• 数组作业合并成一行：几十上百个元素显示成一条分段进度条——绿色完成、红色失败、蓝色运行中、灰色排队，点开能看每个正在运行的元素。 Job arrays in one row: dozens or hundreds of elements become a single segmented bar — green done, red failed, blue running, grey pending — and you can open each running element.
• 排队原因说人话：Priority、AssocGrpBillingRunMinutes、DependencyNeverSatisfied 这类原因翻译成一句话；永远不会开始、需要你动手的那种会标红。 Pending reasons in plain words: Priority, AssocGrpBillingRunMinutes, DependencyNeverSatisfied and friends become one sentence; the ones that will never start without your help are flagged in red.
• 分区空闲资源：点卡片上的 ▦，按需查询各分区还有多少空闲的 CPU、GPU 和节点，帮你决定往哪儿投。 Free resources per partition: click ▦ on the card to see free CPUs, GPUs and nodes per partition, on demand — handy for deciding where to submit.
• 最近 24 小时：有 sacct 时用服务器上的完整记录（数组元素合并成一条），没有时用 ServerIsland 自己观察到的记录。 Last 24 hours: the full record from sacct when available (array elements merged), otherwise what ServerIsland saw itself.
• PBS 也一样：PBS Pro、OpenPBS 和 Torque 用 qselect + qstat -f 读取，数组作业、排队原因（来自 comment）、结束状态（qstat -x）、按住取消（qdel）都有；分区空闲资源只有 Slurm 才有。 PBS too: PBS Pro, OpenPBS and Torque are read with qselect + qstat -f — arrays, pending reasons (from the comment), final states (qstat -x) and hold-to-cancel (qdel) all work; free partition resources are Slurm only.
• 只看某些目录：每台服务器可以只显示工作目录在指定文件夹里的作业和进程，比如只看 ~/projects/LLZO。 Only some folders: limit a server to jobs and processes whose working directory is inside given folders, e.g. just ~/projects/LLZO.
• 单机 Slurm 工作站：可以同时显示作业和直接运行的程序，属于作业的进程不会重复列出。 Single-machine Slurm: show jobs and directly started programs together; processes that belong to a job aren't listed twice.

直连服务器与 GPUDirect servers & GPUs

• 进程：列出你的计算进程（Python、torchrun、计算软件……），系统服务、桌面、编辑器后台这些自动排除；"包含 / 排除"规则可以自己改。占用 GPU 的进程总会显示。 Processes: your compute processes (Python, torchrun, simulation codes…), with system services, desktop and editor helpers filtered out — the include / exclude rules are editable. Anything using a GPU always shows.
• GPU：一两张卡时显示利用率、显存、温度；更多卡时每张一个小方块，显存快满时顶上一条琥珀色的线。 GPUs: utilisation, memory and temperature for one or two cards; with more, one small cell per GPU, with an amber line when its memory is nearly full.
• 监听进程：点卡片上的 👁，从高负载进程里挑一个"监听"，或者在设置里输入 PID。监听的进程不受过滤规则影响，结束时一定提醒；PID 被别的进程复用时也能认出来。 Watch a process: pick one with 👁 on the card or enter a PID in Settings. Watched processes ignore the filters and always alert when they end — and PID reuse is detected.
• 磁盘：用量超过阈值（默认 90%）时提醒一次，回落之后再超过才会再提醒。 Disks: one alert when usage passes the threshold (90% by default), and again only after it drops and rises again.

计算软件识别Simulation software

在任务的工作目录里找特征文件，只读文件末尾或很小的文件，和任务列表在同一次刷新里完成。任何一处和预期不一致，就不显示进度。 ServerIsland looks for telltale files in each job's working directory, reads only small files or file tails, and does it in the same refresh as the job list. If anything doesn't match what it expects, it shows no progress at all.

进度长时间没有变化（超过 10 分钟或 5 个刷新周期）时，任务行上会提示"⚠ 没有新输出"，方便发现卡住的任务。任务结束时会再读一次输出文件的最终状态，避免"最后一次采样还没到目标"被误判为失败。 If the output hasn't changed for a while (10 minutes or five refreshes), the row warns "⚠ no output" so stuck runs stand out. When a job ends, the output is read once more so a run that finished between two refreshes isn't reported as failed.

自定义规则Custom rules

你自己的程序也能显示进度：在 设置 → 进度识别 里添加规则，填一个文件名（可以用 * 通配，取最新的一个）和一个关键字或正则。比如关键字 epoch 能匹配 "Epoch 37/100"、"epoch: 37"；写成 37/100 时会顺便取到目标值。目标值也可以写死，或者从另一个文件里读（比如配置文件里的 max_epochs）。再填一个"结束标记"，进程结束时就能判断是正常完成还是提前退出。 Make your own code show progress: add a rule under Settings → Progress Detection with a file name (wildcards allowed — the newest match is used) and a keyword or regex. The keyword epoch matches "Epoch 37/100" or "epoch: 37", and the "/100" form also gives the target. The target can also be fixed or read from another file (say max_epochs in a config). Add an end marker and ServerIsland can tell a clean finish from an early exit.

规则编辑页有"测试"按钮：选一台服务器和一个任务目录，只读地跑一次，告诉你取到了什么，没取到时显示文件最后几行。 The rule editor has a Test button: pick a server and a job directory, and it runs once, read-only, showing what it found — or the last lines of the file when it found nothing.

脚本也可以主动上报：往 ~/.serverisland/progress/名称.json 写一行 JSON（progress 或 current / total、message、jobid、status），带 jobid 的会合并到对应作业上。 Scripts can also report directly by writing one line of JSON to ~/.serverisland/progress/name.json (progress or current / total, message, jobid, status); entries with a jobid are merged into that job.

提醒Alerts

详情、日志、取消Details, logs & cancelling

• 详情：Slurm 作业来自 scontrol（结束太久的用 sacct），进程来自 /proc：状态、时间、节点、资源、工作目录、完整命令。 Details from scontrol (or sacct for long-finished jobs) and /proc: state, times, nodes, resources, working directory, full command.
• 日志末尾：自动找到作业的 StdOut / StdErr，或进程输出重定向到的文件，每 4 秒刷新最后 40 行；报错行标红，警告标黄。输出直接打到终端（tmux / screen）时会告诉你为什么读不到。 Log tail: finds the job's StdOut / StdErr or the file a process writes to and refreshes the last 40 lines every 4 s, with errors in red and warnings in amber. If output goes to a terminal (tmux / screen), it tells you why there's nothing to read.
• 一键复制：作业号、路径、完整命令，以及一条可以在终端里实时看日志的 ssh … tail -f 命令；还可以直接在终端里 ssh 到工作目录。 One-click copies of IDs, paths, full commands and a ready ssh … tail -f command — or open a terminal straight into the working directory.
• 取消：默认关闭，按服务器开启；按住 1.5 秒 → 再确认一次（写明要执行的命令）→ 服务器上核对任务名称、所有者和启动时间，对不上就不执行。进程先发 TERM，没退出可以再强制结束。 Cancel: off by default, enabled per server; hold 1.5 s → confirm (the exact command is shown) → the server re-checks name, owner and start time and refuses on any mismatch. Processes get TERM first, with a force-kill if they don't exit.

每台服务器的设置Per-server control

Tako

刘海里的像素小章鱼，心情跟着服务器走：The pixel octopus in the notch mirrors your servers:

三个平台Three platforms

设计原则Principles