[petsc-users] Load balancing problem with DMDA

[Xiao, Jianjun (IKET)]

Hello,


We have a finite volume code with structured grid. We use the DMDA data structure.


Now we have the load balancing problem as shown in the attached picture. The domain is decomposed into 4 partitions. The sub-domain in black (proc 2) is solid which is of no interest. We would like to calculate only the fluid part in proc 1, 3 and 4 which means the resource in proc 2 is wasted. Does DMDA now support multi-block with communication between blocks? I mean I build block 1 and assign to proc1, block 2 to proc 3 and block 3 to proc 4. So the solid sub-domain does not exist. Any suggestion?


Or I have to do it using DMPLEX? Since data structure of the whole code is based on DMDA, is there any "simple way" to switch from DMDA to DMPLEX without changing the loop structure in the code?


Thank you.


Best regards

Jianjun

[Barry Smith]

> On Oct 14, 2015, at 3:11 PM, Xiao, Jianjun (IKET) <***@kit.edu> wrote:
>
> Hello,
>
> We have a finite volume code with structured grid. We use the DMDA data structure.
>
> Now we have the load balancing problem as shown in the attached picture. The domain is decomposed into 4 partitions. The sub-domain in black (proc 2) is solid which is of no interest. We would like to calculate only the fluid part in proc 1, 3 and 4 which means the resource in proc 2 is wasted. Does DMDA now support multi-block with communication between blocks? I mean I build block 1 and assign to proc1, block 2 to proc 3 and block 3 to proc 4. So the solid sub-domain does not exist. Any suggestion?

No

>
> Or I have to do it using DMPLEX? Since data structure of the whole code is based on DMDA, is there any "simple way" to switch from DMDA to DMPLEX without changing the loop structure in the code?

No

DMDA is for simple structured grids.

>
> Thank you.
>
> Best regards
> Jianjun
> <pic.JPG>

[Matthew Knepley]

On Wed, Oct 14, 2015 at 3:32 PM, Barry Smith <***@mcs.anl.gov> wrote:

>
> > On Oct 14, 2015, at 3:11 PM, Xiao, Jianjun (IKET) <***@kit.edu>
> wrote:
> >
> > Hello,
> >
> > We have a finite volume code with structured grid. We use the DMDA data
> structure.
> >
> > Now we have the load balancing problem as shown in the attached picture.
> The domain is decomposed into 4 partitions. The sub-domain in black (proc
> 2) is solid which is of no interest. We would like to calculate only the
> fluid part in proc 1, 3 and 4 which means the resource in proc 2 is wasted.
> Does DMDA now support multi-block with communication between blocks? I mean
> I build block 1 and assign to proc1, block 2 to proc 3 and block 3 to proc
> 4. So the solid sub-domain does not exist. Any suggestion?
>
> No
>
> >
> > Or I have to do it using DMPLEX? Since data structure of the whole code
> is based on DMDA, is there any "simple way" to switch from DMDA to DMPLEX
> without changing the loop structure in the code?
>
> No
>
> DMDA is for simple structured grids.


Using Plex would be some work since no one has ever used it to do FD. In
fact, now I think the way to do this
is to only represent the vertices and edges. I would help you if you want
to experiment with that.

Thanks,

Matt


>
> >
> > Thank you.
> >
> > Best regards
> > Jianjun
> > <pic.JPG>
>
>


--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

[Xiao, Jianjun (IKET)]

Matt,


Yes. I would like to try. Where should I start?


________________________________
From: Matthew Knepley <***@gmail.com>
Sent: Thursday, October 15, 2015 3:43 PM
To: Barry Smith
Cc: Xiao, Jianjun (IKET); petsc-***@mcs.anl.gov
Subject: Re: [petsc-users] Load balancing problem with DMDA

On Wed, Oct 14, 2015 at 3:32 PM, Barry Smith <***@mcs.anl.gov<mailto:***@mcs.anl.gov>> wrote:

> On Oct 14, 2015, at 3:11 PM, Xiao, Jianjun (IKET) <***@kit.edu<mailto:***@kit.edu>> wrote:
>
> Hello,
>
> We have a finite volume code with structured grid. We use the DMDA data structure.
>
> Now we have the load balancing problem as shown in the attached picture. The domain is decomposed into 4 partitions. The sub-domain in black (proc 2) is solid which is of no interest. We would like to calculate only the fluid part in proc 1, 3 and 4 which means the resource in proc 2 is wasted. Does DMDA now support multi-block with communication between blocks? I mean I build block 1 and assign to proc1, block 2 to proc 3 and block 3 to proc 4. So the solid sub-domain does not exist. Any suggestion?

No

>
> Or I have to do it using DMPLEX? Since data structure of the whole code is based on DMDA, is there any "simple way" to switch from DMDA to DMPLEX without changing the loop structure in the code?

No

DMDA is for simple structured grids.

Using Plex would be some work since no one has ever used it to do FD. In fact, now I think the way to do this
is to only represent the vertices and edges. I would help you if you want to experiment with that.

Thanks,

Matt


>
> Thank you.
>
> Best regards
> Jianjun
> <pic.JPG>




--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

[Matthew Knepley]

On Thu, Oct 15, 2015 at 8:55 AM, Xiao, Jianjun (IKET) <***@kit.edu>
wrote:

> Matt,
>
>
> Yes. I would like to try. Where should I start?
>
I would take a small 2D problem that you care about and try to make an
equivalent Plex code to the DMDA code.

I think it will be slower than DMDA, but trying a simple thing does not
hurt.

I can make you a simple 2D lattice in Plex.

Thanks,

Matt



> ------------------------------
> *From:* Matthew Knepley <***@gmail.com>
> *Sent:* Thursday, October 15, 2015 3:43 PM
> *To:* Barry Smith
> *Cc:* Xiao, Jianjun (IKET); petsc-***@mcs.anl.gov
> *Subject:* Re: [petsc-users] Load balancing problem with DMDA
>
> On Wed, Oct 14, 2015 at 3:32 PM, Barry Smith <***@mcs.anl.gov> wrote:
>
>>
>> > On Oct 14, 2015, at 3:11 PM, Xiao, Jianjun (IKET) <***@kit.edu>
>> wrote:
>> >
>> > Hello,
>> >
>> > We have a finite volume code with structured grid. We use the DMDA data
>> structure.
>> >
>> > Now we have the load balancing problem as shown in the attached
>> picture. The domain is decomposed into 4 partitions. The sub-domain in
>> black (proc 2) is solid which is of no interest. We would like to calculate
>> only the fluid part in proc 1, 3 and 4 which means the resource in proc 2
>> is wasted. Does DMDA now support multi-block with communication between
>> blocks? I mean I build block 1 and assign to proc1, block 2 to proc 3 and
>> block 3 to proc 4. So the solid sub-domain does not exist. Any suggestion?
>>
>> No
>>
>> >
>> > Or I have to do it using DMPLEX? Since data structure of the whole
>> code is based on DMDA, is there any "simple way" to switch from DMDA to
>> DMPLEX without changing the loop structure in the code?
>>
>> No
>>
>> DMDA is for simple structured grids.
>
>
> Using Plex would be some work since no one has ever used it to do FD. In
> fact, now I think the way to do this
> is to only represent the vertices and edges. I would help you if you want
> to experiment with that.
>
> Thanks,
>
> Matt
>
>
>>
>> >
>> > Thank you.
>> >
>> > Best regards
>> > Jianjun
>> > <pic.JPG>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener