after disk space freed, jenkins nodes doesn't come back online

Question

I looked at https://unix.stackexchange.com/search?q=jenkins & Jenkins support https://issues.jenkins.io

I didn't find out there how to solve that.

initial state

Nodes & main server had disk space issues, and no pipeline were triggered anymore.

After space issue has been solved nodes are still offline ignoring the new status.

how can I refresh those nodes to online state?

Architecture :

• host connections uses java / git & ssh depending on with step/pipeline is running
• pipelines usually uses git to get last groovy & script files ; & also run the jenkins groovy file on nodes
• steps are calling ssh on either bash or powershell access when needed to access nodes & run scripts.

Done :

• cleaned /var/lib on main jenkins server

  find . -name "xxxx" -atime +365 -exec rm {} \;

• cleaned application drives on each nodes

    $limit = (Get-Date).AddDays(-30)
    $path = "xxxxxxx"
    Get-ChildItem -Path $path -Recurse -Force -filter XXXXX | Where-Object { !$_.PSIsContainer -and $_.CreationTime -lt $limit } | Remove-Item -Force

• restarted jenkins service on main jenkins server

  sudo systemctl restart jenkins

Current Status :

• Those nodes remains offline (and still display as error the same space amount warning, this is like it doesn't read/detect the new available space)

    Disk space is too low. Only 0.000GB left on F:\jenkins2.
    
    Agent is connected.

on machines for example :

DriveType    : 3
ProviderName :
FreeSpace    : 255938342912
Size         : 449998483456
VolumeName   : Disk2

so jenkins ignore issue about disk space is solved & keeps nods offline how to make them online again ?

Answer 1

Based on your comments and question edits. It appears the space issue isn't with your host RHEL system.

The problem is with your "Windows Images" (I presume you mean virtual machines). The space issue is clearly talking about a Windows path F:\jenkins2 containing a Windows drive letter F: and path backslash path seperator \. If it were a linux path there would be no : in the path and it would use a forward slash / as a path separator.

You've not given an information on what type of VM these are beyond being a "Minimal Windows image", so I can't offer any advice about how to clear down the existing images. To do that you'd probably need to get a command line running inside the image and manually delete files.

What you can do is...

• Confirm that your existing builds are still accessible (and not only stored in the build nodes). To do this, just log into Jenkins and check you can still download build artefacts from Windows builds.
• Destroy your build images and recreate them. Depending on your VM software it might be possible to revert them to an earlier version.

---

I'd also recommend that for future you check your Jenkins options for disk space management. I suspect that your build nodes have filled up with old builds which they do not need to keep.