Jenkins › Jenkins issues

[JIRA] (JENKINS-13837) vSphere Plugin not powering up virtual machines

Classic

List

Threaded

19 messages Options

JIRA noreply@jenkins-ci.org

[JIRA] (JENKINS-13837) vSphere Plugin not powering up virtual machines

Moshe Belostotsky created JENKINS-13837:
-------------------------------------------

Summary: vSphere Plugin not powering up virtual machines
Key: JENKINS-13837
URL: https://issues.jenkins-ci.org/browse/JENKINS-13837
Project: Jenkins
Issue Type: Bug
Components: slave-setup, vsphere-cloud
Affects Versions: current
Environment: Virtualization: VMware vSphere ESXI 5.0.0
Jenkins Server: Ubuntu 11.04, Jenkins-CI 1.464, vSphere Cloud Plugin 0.9
2X Ubuntu slave: 11.04
2X Windows slaves: Win 7

Reporter: Moshe Belostotsky
Assignee: jswager
Priority: Blocker
Attachments: vSphere.png

Steps:
1. Upgrade vSphere Cloud Plugin to 0.9
2. Try to start one of the Virtual slaves using the plugin
3. Watch the vSphere client console
4. Watch the Slave log

Expected:
3. The VM should power up
4. The log should show some out.

Actual:

3. The VM not starting
4. Nothing in the log except of the spinning circle.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

JIRA noreply@jenkins-ci.org

[JIRA] (JENKINS-13837) vSphere Plugin not powering up virtual machines

[ https://issues.jenkins-ci.org/browse/JENKINS-13837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Moshe Belostotsky updated JENKINS-13837:
----------------------------------------

Priority: Critical (was: Blocker)

> vSphere Plugin not powering up virtual machines
> -----------------------------------------------
>
> Key: JENKINS-13837
> URL: https://issues.jenkins-ci.org/browse/JENKINS-13837
> Project: Jenkins
> Issue Type: Bug
> Components: slave-setup, vsphere-cloud
> Affects Versions: current
> Environment: Virtualization: VMware vSphere ESXI 5.0.0
> Jenkins Server: Ubuntu 11.04, Jenkins-CI 1.464, vSphere Cloud Plugin 0.9
> 2X Ubuntu slave: 11.04
> 2X Windows slaves: Win 7
> Reporter: Moshe Belostotsky
> Assignee: jswager
> Priority: Critical
> Labels: slave, vsphere
> Attachments: vSphere.png
>
>
> Steps:
> 1. Upgrade vSphere Cloud Plugin to 0.9
> 2. Try to start one of the Virtual slaves using the plugin
> 3. Watch the vSphere client console
> 4. Watch the Slave log
> Expected:
> 3. The VM should power up
> 4. The log should show some out.
> Actual:
> 3. The VM not starting
> 4. Nothing in the log except of the spinning circle.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

JIRA noreply@jenkins-ci.org

[JIRA] (JENKINS-13837) vSphere Plugin not powering up virtual machines

In reply to this post by JIRA noreply@jenkins-ci.org

This is a problem with Jenkins; it was fixed in 1.468 (or maybe 1.467) and higher. Upgrade Jenkins (and the vSphere Plugin) and things should start working properly.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

JIRA noreply@jenkins-ci.org

[JIRA] (JENKINS-13837) vSphere Plugin not powering up virtual machines

In reply to this post by JIRA noreply@jenkins-ci.org

I have upgraded Jenkins to 1.471 and still same problem.

JIRA noreply@jenkins-ci.org

[JIRA] (JENKINS-13837) vSphere Plugin not powering up virtual machines

In reply to this post by JIRA noreply@jenkins-ci.org

Did you also upgrade the vSphere plugin to v.10? There was code in v0.9 that attempted to work around the Jenkins problems, but ultimately, just made a worse mess of things.

JIRA noreply@jenkins-ci.org

[JIRA] (JENKINS-13837) vSphere Plugin not powering up virtual machines

In reply to this post by JIRA noreply@jenkins-ci.org

Yes, IM on v0.10

JIRA noreply@jenkins-ci.org

[JIRA] (JENKINS-13837) vSphere Plugin not powering up virtual machines

In reply to this post by JIRA noreply@jenkins-ci.org

That takes care of the easy troubleshooting - now for the hard stuff.

Are you trying to start the slave via the slave page (Launch Slave Agent), or are you trying to start the slave by having a job that requires the slave?

Is the slave setup to talk to a VM and optional snapshot properly? In other words, does the "Test VM Connection" pass? Also, if using a snapshot, do you have any other slaves using the same VM with different snapshots?

Do you have "Force VM Launch" enabled? And is "Delay between launch and boot complete" set to a reasonable value?

How is the slave agent set to connect - what is the value(s) fo the "Secondary Launch Method"? What about the "Availability" settings?

Do you see any actions from Jenkins in the vCenter logs?

JIRA noreply@jenkins-ci.org

[JIRA] (JENKINS-13837) vSphere Plugin not powering up virtual machines

In reply to this post by JIRA noreply@jenkins-ci.org

1. I have tried both methods.
2. Test VM connection works ok, I am not using snapshots.
3. "Force VM Launch" is enabled.
Delay set to 60.
4. Availability set to "Take this slave on-line when in demand..."
The secondary launch method is Java Web Start.
5. Yes, I saw some connect and disconnect of user.

Something strange that it sometimes do start machines, But still i cannot c anything related to the vSphere plugin in the slave logs.

JIRA noreply@jenkins-ci.org

[JIRA] (JENKINS-13837) vSphere Plugin not powering up virtual machines

In reply to this post by JIRA noreply@jenkins-ci.org

Change By:	Moshe Belostotsky (26/Jun/12 12:41 PM)
Environment:	Virtualization: VMware vSphere ESXI 5.0.0 Jenkins Server: Ubuntu 11.04, Jenkins-CI 1. 464 472 , vSphere Cloud Plugin 0. 9 10 2X Ubuntu slave: 11.04 2X Windows slaves: Win 7

JIRA noreply@jenkins-ci.org

[JIRA] (JENKINS-13837) vSphere Plugin not powering up virtual machines

In reply to this post by JIRA noreply@jenkins-ci.org

Try increasing the delay. When using Java Web Start, the plugin is expecting that the VM will initiate it's own connection to Jenkins during the delay. The delay time starts at either 1) the presence of VMTools, if that option is selected, or 2) when the machine is powered on.

As for the logs, I'll have to investigate that further. There SHOULD be at least a few vSphere log lines that appear during the startup. For all start methods except Java Web Start, they get wiped out - each of the other start mechanisms starts out by wiping the log lines. But there should have been a few lines present for a bit.

JIRA noreply@jenkins-ci.org

[JIRA] (JENKINS-13837) vSphere Plugin not powering up virtual machines

In reply to this post by JIRA noreply@jenkins-ci.org

Attached by:Srinath Logs collected while launching the vSphere slaves
Change By:	Srinath C (22/Jul/12 4:35 AM)
Attachment:	slave-logs.zip

JIRA noreply@jenkins-ci.org

[JIRA] (JENKINS-13837) vSphere Plugin not powering up virtual machines

In reply to this post by JIRA noreply@jenkins-ci.org

I'm currently facing the same issue.

My setup:
Virtualization: VMware vSphere ESXI 5.0.0
Jenkins Server: Centos 6, Jenkins-CI 1.470, vSphere Cloud Plugin 0.10
Centos slaves: 6.2

Have attached the logs for the slave.

JIRA noreply@jenkins-ci.org

[JIRA] (JENKINS-13837) vSphere Plugin not powering up virtual machines

In reply to this post by JIRA noreply@jenkins-ci.org

I'm currently facing the same issue.

My setup:
Virtualization: VMware vSphere ESXI 5.0.0
Jenkins Server: Centos 6, Jenkins-CI 1.470, vSphere Cloud Plugin 0.10
Centos slaves: 6.2

Have attached the logs for the slave (slave-logs.zip).

JIRA noreply@jenkins-ci.org

[JIRA] (JENKINS-13837) vSphere Plugin not powering up virtual machines

In reply to this post by JIRA noreply@jenkins-ci.org

I'm currently facing the same issue.

My setup:
Virtualization: VMware vSphere ESXI 5.0.0
Jenkins Server: Centos 6, Jenkins-CI 1.470, vSphere Cloud Plugin 0.10
VMs: Centos 6.2, snapshot set to "jenkins_setup"

Have attached the logs for the slave (slave-logs.zip).
Note that I can see the vm being reverted to the configured snapshot but beyond that the slave does not proceed.

JIRA noreply@jenkins-ci.org

[JIRA] (JENKINS-13837) vSphere Plugin not powering up virtual machines

In reply to this post by JIRA noreply@jenkins-ci.org

Logs collected while launching the vSphere slaves (slave-logs.zip)

JIRA noreply@jenkins-ci.org

[JIRA] (JENKINS-13837) vSphere Plugin not powering up virtual machines

In reply to this post by JIRA noreply@jenkins-ci.org

I got the issue resolved. VMs are successfully getting launched on the vSphere server now.
It was not even related to jenkins or the vSphere Cloud plugin.
It was the Leap Second bug on the jenkins server that was causing the problem.

I took a jstack of the jenkins process while the slave was stuck at launching and found that the thread was hung at :
"pool-6-thread-8" daemon prio=10 tid=0x00007f75d8252800 nid=0x7292 sleeping[0x00007f768651c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.vmware.vim25.mo.Task.waitForTask(Task.java:229)
at com.vmware.vim25.mo.Task.waitForTask(Task.java:152)
at org.jenkinsci.plugins.vSphereCloudLauncher.launch(vSphereCloudLauncher.java:169)
at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:200)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)

After spending a lot of time peeking into the source code I realized that even a simple program to sleep for 1 second using Thread.currentThread().sleep(1000) was also getting stuck indefinitely.

Fortunately, I found the reason at http://stackoverflow.com/questions/11294573/thread-sleep-never-returns and a simple reboot of the server solved the problem.

JIRA noreply@jenkins-ci.org

[JIRA] (JENKINS-13837) vSphere Plugin not powering up virtual machines

In reply to this post by JIRA noreply@jenkins-ci.org

Change By:	Jason Swager (17/Oct/12 2:51 PM)
Assignee:	jswager Jason Swager

JIRA noreply@jenkins-ci.org

[JIRA] (JENKINS-13837) vSphere Plugin not powering up virtual machines

In reply to this post by JIRA noreply@jenkins-ci.org

Change By:	Jason Swager (17/Oct/12 2:51 PM)
Status:	Open In Progress

JIRA noreply@jenkins-ci.org

[JIRA] (JENKINS-13837) vSphere Plugin not powering up virtual machines

In reply to this post by JIRA noreply@jenkins-ci.org

Problem was in Java, not in this plugin.
Change By:	Jason Swager (17/Oct/12 2:52 PM)
Status:	In Progress Resolved
Resolution:	Not A Defect