whamcloud/integrated-manager-for-lustre

Jason Williams

@uberlinuxguy

@johnsonw yes, sorry I was interrupted for a bit there.

Will Johnson

@johnsonw

No problem Jason. I’m making progress and will keep you posted.

yguvvala

@yguvvala

@uberlinuxguy just checking on the status.

Will Johnson

@johnsonw

@yguvvala I’m working on a script to resolve some database issues.

Will Johnson

@johnsonw

Hi @uberlinuxguy, I’ve created a script that cleans up the majority of the volumes on the volumes page. Here is a screenshot of my local volume page after running the script:

Let’s sync up tomorrow in the morning and i’ll send you the script.

Jason Williams

@uberlinuxguy

Hi @johnsonw Let me know when you are available. I am in the office now.

Will Johnson

@johnsonw

Hi @uberlinuxguy, the script is able to resolve the majority of the items on the volumes page, but there are four items that only have one entry in the list you sent:

- 600a098000591fd00000012b5527440d - scratch OST003d - only has one entry
- 600a09800060f4860000059e55274536 - scratch OST0043 - only has one entry
- 600a0980006355d50000050155274407 - scratch-OST0045 - only has one entry
- 600a0980006355d50000050955274432 - scratch OST0047 - only has one entry

Since there is only 1 entry it can’t create an ha pair.

Jason Williams

@uberlinuxguy

which list was that in?

Will Johnson

@johnsonw

https://www.marcc.jhu.edu/downloads/lustre/targets_readout.txt

Do a search for “ost003d” for example and you will see there is only 1 entry.

Jason Williams

@uberlinuxguy

ah because oss12 is offline

Will Johnson

@johnsonw

ah ok that makes sense

Jason Williams

@uberlinuxguy

all of those seem to come from oss11/oss12

Will Johnson

@johnsonw

Yes

Do you know if you will be able to get oss12 back up?

In either case, I can still send the script your way and it will resolve the majority of the items in the list.

Jason Williams

@uberlinuxguy

I am going to try to boot oss12 now

Will Johnson

@johnsonw

Ok. Also, please make sure you backup the database before running the script.

Do you want me to e-mail the script or is it ok to paste into this chat?

Jason Williams

@uberlinuxguy

I can just re-run the same dump command I run to send you the db right?

Will Johnson

@johnsonw

Yes.

Jason Williams

@uberlinuxguy

you can paste it into chat I suppose

Will Johnson

@johnsonw

I can also put it on the ticket. That’s probably better. Is that ok?

Jason Williams

@uberlinuxguy

Yeah sure

Will Johnson

@johnsonw

Script posted intel-hpdd/intel-manager-for-lustre#637.

Jason Williams

@uberlinuxguy

can I add the targets that should be on oss12 to the target_list variable before I run this?

Will Johnson

@johnsonw

It should be fine but just keep in mind that it hasn’t been tested.

Jason Williams

@uberlinuxguy

Jason Williams

@uberlinuxguy

awesome that cleared the volumes page.

and the number of volumes per host is now 12

Will Johnson

@johnsonw

Excellent news :clap:

And failover still works correct?

Jason Williams

@uberlinuxguy

failing a couple of the targets back to their primary hosts and that seems to be working too.

Will Johnson

@johnsonw

awesome :+1:

Jason Williams

@uberlinuxguy

Things are looking much better now.

Migration works, everything is reporting right, I think we are safe to close the currently open ticket for the upgrade.

Did you get a change to find out anything about the striping and the handful of jobs taking down the OST?

Will Johnson

@johnsonw

Great to hear @uberlinuxguy. I don’t have any information on the striping yet but will let you know as soon as I do.

Jason Williams

@uberlinuxguy

Ok sounds good.

Thank you so much @yguvvala, @johnsonw, and @jgrund for all your help

Joe Grund

@jgrund

@uberlinuxguy glad to hear you’re in a good spot with the upgrade. Gentle reminder to please open tickets on issues discussed earlier (obsoleting apache etc…)

Jason Williams

@uberlinuxguy

Oh yes.

I have those in a text pad somewhere...

Will Johnson

@johnsonw

@uberlinuxguy I just got a reponse on the striping information:

Depending on the number of OSTs, I’ve seen some of the big labs use 4 as a default stripe for the filesystem.

Other than user education, there’s not much you can do by force.  Maybe OST-pools, but that doesn’t prevent someone from hammering a single OST.

Jason Williams

@uberlinuxguy

I guess the other question would be is there a way to update striping for existing files. I believe the answer is no.... cuz how would you do that, but figured it's worth asking.

Where communities thrive

People

Repo info

Activity