We need a fallback mirror that's always in sync

[kohsuke Kawaguchi (CB)]

Jenkins handles a lot of download requests for core as well as plugins.
So we rely on our mirrors to actually deliver bits.

Unfortunately, by the nature of our mirrors, there's some time lag
between the time we stage our master copy of files (which gets created
as a part of the update center metadata generation process) to the time
mirrors pick up those files.

But as soon as bits are staged in the master copy, those URLs get
advertised (download link from http://jenkins-ci.org/ and update center
metadata.) Right now, download requests that happened during this window
will result in 404.

What can we do to fix this? Does anyone have any thoughts?


We used to serve binaries from our master server in this situation
without forwarding requests to mirrors, but this was the root cause why
we ended up with the $5000 hole in our budget last year. So naturally I
suspect Tyler will be afraid of enabling this again.

If someone is willing to be a mirror that allows us to push files,
that'd be ideal, as we can ensure that there's always at least one
mirror that has all the copies all the time. We can set the priority
very low to make sure that it only gets used very rarely.

Another possibility is maybe to run some kind of bandwidth-restricted
HTTP server. In that way, maybe we can convince ourselves that even in
the worst case we won't go bankrupt.

The third possibility might be to buy a bigger disk to
cabbage.jenkins-ci.org. My understanding is that this machine is in
OSUOSL, so we can use it to serve bytes without worrying about incurring
overcharge.


BTW, right now, we have about 58GB of data that needs to be mirrored.

--
Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/
Try Nectar, our professional version of Jenkins

[Lloyd Chang]

Kohsuke asked, "What can we do to fix this? Does anyone have any thoughts?"

Proposal with steps

1. Rename host jenkins-ci.org to master.jenkins-ci.org

2. Change jenkins-ci.org to a CNAME (if not already CNAME)

3. Set jenkins-ci.org CNAME to master.jenkins-ci.org

4. Before staging job, set jenkins-ci.org CNAMfE to mirrors.jenkins-ci.org (or cucumber.jenkins-ci.org, etc.)

5. mirrors.jenkins-ci.org push notification to staging job process, or staging job process pull status from mirrors.jenkins-ci.org

6. After staging job recognizes that  mirrors have pick up files from master, set jenkins-ci.org CNAME back to master.jenkins-ci.org

Thanks,

Lloyd

On Fri, Sep 7, 2012 at 7:05 PM, Kohsuke Kawaguchi <[hidden email]> wrote:

Jenkins handles a lot of download requests for core as well as plugins. So we rely on our mirrors to actually deliver bits.

Unfortunately, by the nature of our mirrors, there's some time lag between the time we stage our master copy of files (which gets created as a part of the update center metadata generation process) to the time mirrors pick up those files.

But as soon as bits are staged in the master copy, those URLs get advertised (download link from http://jenkins-ci.org/ and update center metadata.) Right now, download requests that happened during this window will result in 404.

What can we do to fix this? Does anyone have any thoughts?


We used to serve binaries from our master server in this situation without forwarding requests to mirrors, but this was the root cause why we ended up with the $5000 hole in our budget last year. So naturally I suspect Tyler will be afraid of enabling this again.

If someone is willing to be a mirror that allows us to push files, that'd be ideal, as we can ensure that there's always at least one mirror that has all the copies all the time. We can set the priority very low to make sure that it only gets used very rarely.

Another possibility is maybe to run some kind of bandwidth-restricted HTTP server. In that way, maybe we can convince ourselves that even in the worst case we won't go bankrupt.

The third possibility might be to buy a bigger disk to cabbage.jenkins-ci.org. My understanding is that this machine is in OSUOSL, so we can use it to serve bytes without worrying about incurring overcharge.


BTW, right now, we have about 58GB of data that needs to be mirrored.

--
Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/
Try Nectar, our professional version of Jenkins

[R. Tyler Croy]

On Fri, 07 Sep 2012, Kohsuke Kawaguchi wrote:

>
> Jenkins handles a lot of download requests for core as well as
> plugins. So we rely on our mirrors to actually deliver bits.
>
> Unfortunately, by the nature of our mirrors, there's some time lag
> between the time we stage our master copy of files (which gets
> created as a part of the update center metadata generation process)
> to the time mirrors pick up those files.
>
> But as soon as bits are staged in the master copy, those URLs get
> advertised (download link from http://jenkins-ci.org/ and update
> center metadata.) Right now, download requests that happened during
> this window will result in 404.
>
> What can we do to fix this? Does anyone have any thoughts?

Technically the OSUOSL mirror system is one we push to, I think we try to only
run rsync's on a poll based interval but there's no reason we can't "push"
more.


Both cabbage and lettuce have LVM-managed disks, so we should be able expand
those volumes if needed. MirrorBrain allows for setting a fallback mirror which
we currently set to the ftp-nyc OSUOSL mirror (IIRC). We can easily set that to
cabbage or lettuce if we chose those as a dedicated mirror master.

- R. Tyler Croy
--------------------------------------
    Code: https://github.com/rtyler
 Chatter: https://twitter.com/agentdero

[Kohsuke Kawaguchi]

I believe DNS propagation delay makes this approach unworkable.

2012/9/9 Lloyd Chang <[hidden email]>:

> Kohsuke asked, "What can we do to fix this? Does anyone have any thoughts?"
>
> Proposal with steps
> 1. Rename host jenkins-ci.org to master.jenkins-ci.org
> 2. Change jenkins-ci.org to a CNAME (if not already CNAME)
> 3. Set jenkins-ci.org CNAME to master.jenkins-ci.org
> 4. Before staging job, set jenkins-ci.org CNAMfE to mirrors.jenkins-ci.org
> (or cucumber.jenkins-ci.org, etc.)
> 5. mirrors.jenkins-ci.org push notification to staging job process, or
> staging job process pull status from mirrors.jenkins-ci.org
> 6. After staging job recognizes that  mirrors have pick up files from
> master, set jenkins-ci.org CNAME back to master.jenkins-ci.org
>
> Thanks,
> Lloyd
>
> On Fri, Sep 7, 2012 at 7:05 PM, Kohsuke Kawaguchi <[hidden email]>
> wrote:
>>
>>
>> Jenkins handles a lot of download requests for core as well as plugins. So
>> we rely on our mirrors to actually deliver bits.
>>
>> Unfortunately, by the nature of our mirrors, there's some time lag between
>> the time we stage our master copy of files (which gets created as a part of
>> the update center metadata generation process) to the time mirrors pick up
>> those files.
>>
>> But as soon as bits are staged in the master copy, those URLs get
>> advertised (download link from http://jenkins-ci.org/ and update center
>> metadata.) Right now, download requests that happened during this window
>> will result in 404.
>>
>> What can we do to fix this? Does anyone have any thoughts?
>>
>>
>> We used to serve binaries from our master server in this situation without
>> forwarding requests to mirrors, but this was the root cause why we ended up
>> with the $5000 hole in our budget last year. So naturally I suspect Tyler
>> will be afraid of enabling this again.
>>
>> If someone is willing to be a mirror that allows us to push files, that'd
>> be ideal, as we can ensure that there's always at least one mirror that has
>> all the copies all the time. We can set the priority very low to make sure
>> that it only gets used very rarely.
>>
>> Another possibility is maybe to run some kind of bandwidth-restricted HTTP
>> server. In that way, maybe we can convince ourselves that even in the worst
>> case we won't go bankrupt.
>>
>> The third possibility might be to buy a bigger disk to
>> cabbage.jenkins-ci.org. My understanding is that this machine is in OSUOSL,
>> so we can use it to serve bytes without worrying about incurring overcharge.
>>
>>
>> BTW, right now, we have about 58GB of data that needs to be mirrored.
>>
>> --
>> Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/
>> Try Nectar, our professional version of Jenkins
>
>

--
Kohsuke Kawaguchi

[Kohsuke Kawaguchi]

2012/9/9 R. Tyler Croy <[hidden email]>:

>
> On Fri, 07 Sep 2012, Kohsuke Kawaguchi wrote:
>
>>
>> Jenkins handles a lot of download requests for core as well as
>> plugins. So we rely on our mirrors to actually deliver bits.
>>
>> Unfortunately, by the nature of our mirrors, there's some time lag
>> between the time we stage our master copy of files (which gets
>> created as a part of the update center metadata generation process)
>> to the time mirrors pick up those files.
>>
>> But as soon as bits are staged in the master copy, those URLs get
>> advertised (download link from http://jenkins-ci.org/ and update
>> center metadata.) Right now, download requests that happened during
>> this window will result in 404.
>>
>> What can we do to fix this? Does anyone have any thoughts?
>
>
> Technically the OSUOSL mirror system is one we push to, I think we try to only
> run rsync's on a poll based interval but there's no reason we can't "push"
> more.

If I remember correctly, the actual push to OSUOSL mirror system is
still asynchronous --- we rsync files to the staging area, then run a
script, which pokes their system that eventually delivers bits to the
actual mirror servers.

So we still have no way of knowing when files actually become
downloadable via mirrors.

We can rework the way files are generated (by pushing files first then
page scrape apache index or check with mirrorbrain to figure out what
files are downloadable), but I think it's easier to buy a disk and
have a fallback that we can control synchronously.

> Both cabbage and lettuce have LVM-managed disks, so we should be able expand
> those volumes if needed. MirrorBrain allows for setting a fallback mirror which
> we currently set to the ftp-nyc OSUOSL mirror (IIRC). We can easily set that to
> cabbage or lettuce if we chose those as a dedicated mirror master.

OK, this is good to know. What is the process involved in getting a
disk added? I guess I should ask in #osuosl? Do you know what
interface these servers take?

... actually, you know what? looks like cabbage has a 800GB VG that's
pretty much unused! So there's no need for additional disk.

Should I create a LV out of it and mount it to /srv like we do elsewhere?

> - R. Tyler Croy
> --------------------------------------
>     Code: https://github.com/rtyler
>  Chatter: https://twitter.com/agentdero



--
Kohsuke Kawaguchi

[Lloyd Chang]

If we can't reliably force flush DNS propagation on client sides, then another option's a dedicated HTTP that redirects.

Per your conversation with R. Tyler Croy, it seems like we'd have to poll for status of rsync from mirrors in order to change the HTTP redirect from master back to mirror.

Alternatively, perhaps continue pursuing the route of expanding storage, then you can transparently rotate volume snapshots (before and after rsync finishes) using a cluster or high availability failover via LVM.

Another option's using a snapshot-ready file system like BTRFS, then you wouldn't need LVM.

Cheers,

Lloyd

On Sun, Sep 9, 2012 at 2:23 PM, Kohsuke Kawaguchi <[hidden email]> wrote:

I believe DNS propagation delay makes this approach unworkable.

2012/9/9 Lloyd Chang <[hidden email]>:

> Kohsuke asked, "What can we do to fix this? Does anyone have any thoughts?"
>
> Proposal with steps
> 1. Rename host jenkins-ci.org to master.jenkins-ci.org
> 2. Change jenkins-ci.org to a CNAME (if not already CNAME)
> 3. Set jenkins-ci.org CNAME to master.jenkins-ci.org
> 4. Before staging job, set jenkins-ci.org CNAMfE to mirrors.jenkins-ci.org
> (or cucumber.jenkins-ci.org, etc.)
> 5. mirrors.jenkins-ci.org push notification to staging job process, or
> staging job process pull status from mirrors.jenkins-ci.org
> 6. After staging job recognizes that  mirrors have pick up files from
> master, set jenkins-ci.org CNAME back to master.jenkins-ci.org
>
> Thanks,
> Lloyd
>
> On Fri, Sep 7, 2012 at 7:05 PM, Kohsuke Kawaguchi <[hidden email]>
> wrote:
>>
>>
>> Jenkins handles a lot of download requests for core as well as plugins. So
>> we rely on our mirrors to actually deliver bits.
>>
>> Unfortunately, by the nature of our mirrors, there's some time lag between
>> the time we stage our master copy of files (which gets created as a part of
>> the update center metadata generation process) to the time mirrors pick up
>> those files.
>>
>> But as soon as bits are staged in the master copy, those URLs get
>> advertised (download link from http://jenkins-ci.org/ and update center
>> metadata.) Right now, download requests that happened during this window
>> will result in 404.
>>
>> What can we do to fix this? Does anyone have any thoughts?
>>
>>
>> We used to serve binaries from our master server in this situation without
>> forwarding requests to mirrors, but this was the root cause why we ended up
>> with the $5000 hole in our budget last year. So naturally I suspect Tyler
>> will be afraid of enabling this again.
>>
>> If someone is willing to be a mirror that allows us to push files, that'd
>> be ideal, as we can ensure that there's always at least one mirror that has
>> all the copies all the time. We can set the priority very low to make sure
>> that it only gets used very rarely.
>>
>> Another possibility is maybe to run some kind of bandwidth-restricted HTTP
>> server. In that way, maybe we can convince ourselves that even in the worst
>> case we won't go bankrupt.
>>
>> The third possibility might be to buy a bigger disk to
>> cabbage.jenkins-ci.org. My understanding is that this machine is in OSUOSL,
>> so we can use it to serve bytes without worrying about incurring overcharge.
>>
>>
>> BTW, right now, we have about 58GB of data that needs to be mirrored.
>>
>> --
>> Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/
>> Try Nectar, our professional version of Jenkins
>
>

--
Kohsuke Kawaguchi

[Andrew Melo]

On Sun, Sep 9, 2012 at 10:14 PM, Lloyd Chang <[hidden email]> wrote:

If we can't reliably force flush DNS propagation on client sides, then another option's a dedicated HTTP that redirects.

Per your conversation with R. Tyler Croy, it seems like we'd have to poll for status of rsync from mirrors in order to change the HTTP redirect from master back to mirror.

Alternatively, perhaps continue pursuing the route of expanding storage, then you can transparently rotate volume snapshots (before and after rsync finishes) using a cluster or high availability failover via LVM.

Another option's using a snapshot-ready file system like BTRFS, then you wouldn't need LVM.

It adds a bit more latency, but couldn't the metadata change be delayed until all the mirrors report back that they have completed the sync

 

Cheers,

Lloyd

On Sun, Sep 9, 2012 at 2:23 PM, Kohsuke Kawaguchi <[hidden email]> wrote:

I believe DNS propagation delay makes this approach unworkable.

2012/9/9 Lloyd Chang <[hidden email]>:

> Kohsuke asked, "What can we do to fix this? Does anyone have any thoughts?"
>
> Proposal with steps
> 1. Rename host jenkins-ci.org to master.jenkins-ci.org
> 2. Change jenkins-ci.org to a CNAME (if not already CNAME)
> 3. Set jenkins-ci.org CNAME to master.jenkins-ci.org
> 4. Before staging job, set jenkins-ci.org CNAMfE to mirrors.jenkins-ci.org
> (or cucumber.jenkins-ci.org, etc.)
> 5. mirrors.jenkins-ci.org push notification to staging job process, or
> staging job process pull status from mirrors.jenkins-ci.org
> 6. After staging job recognizes that  mirrors have pick up files from
> master, set jenkins-ci.org CNAME back to master.jenkins-ci.org
>
> Thanks,
> Lloyd
>
> On Fri, Sep 7, 2012 at 7:05 PM, Kohsuke Kawaguchi <[hidden email]>
> wrote:
>>
>>
>> Jenkins handles a lot of download requests for core as well as plugins. So
>> we rely on our mirrors to actually deliver bits.
>>
>> Unfortunately, by the nature of our mirrors, there's some time lag between
>> the time we stage our master copy of files (which gets created as a part of
>> the update center metadata generation process) to the time mirrors pick up
>> those files.
>>
>> But as soon as bits are staged in the master copy, those URLs get
>> advertised (download link from http://jenkins-ci.org/ and update center
>> metadata.) Right now, download requests that happened during this window
>> will result in 404.
>>
>> What can we do to fix this? Does anyone have any thoughts?
>>
>>
>> We used to serve binaries from our master server in this situation without
>> forwarding requests to mirrors, but this was the root cause why we ended up
>> with the $5000 hole in our budget last year. So naturally I suspect Tyler
>> will be afraid of enabling this again.
>>
>> If someone is willing to be a mirror that allows us to push files, that'd
>> be ideal, as we can ensure that there's always at least one mirror that has
>> all the copies all the time. We can set the priority very low to make sure
>> that it only gets used very rarely.
>>
>> Another possibility is maybe to run some kind of bandwidth-restricted HTTP
>> server. In that way, maybe we can convince ourselves that even in the worst
>> case we won't go bankrupt.
>>
>> The third possibility might be to buy a bigger disk to
>> cabbage.jenkins-ci.org. My understanding is that this machine is in OSUOSL,
>> so we can use it to serve bytes without worrying about incurring overcharge.
>>
>>
>> BTW, right now, we have about 58GB of data that needs to be mirrored.
>>
>> --
>> Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/
>> Try Nectar, our professional version of Jenkins
>
>

--
Kohsuke Kawaguchi

--
--
Andrew Melo