[petsc-users] MatGetSubMatrices or VecCreateGhost

[Bikash Kanungo]

Hi,

I need to multiply a matrix A with several vectors v_i's and use the
resultant vectors in my calculation which involves access to non-local
nodes. I can think of two possible ways to do it:

1. Store the vectors v_i's into an MPIDENSE matrix and perform MatMatMult
with A to get a resultant dense matrix B. I can now use MatGetSubMatrices
on B to access the non-local part required in my subsequent calculations.

2. Use MatMult to evaluate A*v_i for each of the vectors and provide ghost
padding through VecCreateGhost for the resultant vector.

Which of the two is a more efficient way to perform the task? Also, if
there is a better way possible to achieve the same, then kindly let me know.

Thanks,
Bikash

[Barry Smith]

Hi,
I need to multiply a matrix A with several vectors v_i's and use the resultant vectors in my calculation which involves access to non-local nodes.

What percentage of the "non-local" results do you need in "your calculation", a few percent, all of them? How do you know which "non-local" results you will need for "your calculation"? Are they ghost points on a grid? Something else?

What is "several vectors"? 5? 100? 1000?

Barry

1. Store the vectors v_i's into an MPIDENSE matrix and perform MatMatMult with A to get a resultant dense matrix B. I can now use MatGetSubMatrices on B to access the non-local part required in my subsequent calculations.
2. Use MatMult to evaluate A*v_i for each of the vectors and provide ghost padding through VecCreateGhost for the resultant vector.
Which of the two is a more efficient way to perform the task? Also, if there is a better way possible to achieve the same, then kindly let me know.
Thanks,
Bikash
--
Bikash S. Kanungo
PhD Student
Computational Materials Physics Group
Mechanical Engineering
University of Michigan

[Bikash Kanungo]

Hi Barry,

I'm sorry, I should have provided more details.

Matrix A defined in my problem is of size NxN and I have N vectors v_i,
each of length N, which are to be multiplied by A. N is fairly small
10000-50000. Ideally speaking, it doesn't require parallel storage. But I
will soon exhaust memory by storing both A and v_i's sequentially. So I
planned to implement it through a sub-communiator as this Mat-Vec(s)
operation(s) is part of a bigger problem which requires a greater degree of
parallelization.

My ghost points are physical grid points, FEM nodal points to be exact. So
I have a list of nodes whose data is required on a given processor to
perform evaluation of any physical quantity. It is through this list that I
decide the ghost points (I check if the global ids of these nodes are
within the ownership range for the PETSc partioning or not).

Now depending upon the number of processors I put into my sub-communicator,
the percentage of non-local data I'll require will range from 5-10%.

I hope I'm able to put forth my problem properly and adequately.

Thanks,
Bikash

Post by Bikash Kanungo

Post by Bikash Kanungo
Hi,
I need to multiply a matrix A with several vectors v_i's and use the

resultant vectors in my calculation which involves access to non-local
nodes.
What percentage of the "non-local" results do you need in "your
calculation", a few percent, all of them? How do you know which "non-local"
results you will need for "your calculation"? Are they ghost points on a
grid? Something else?
What is "several vectors"? 5? 100? 1000?
Barry

Post by Bikash Kanungo
1. Store the vectors v_i's into an MPIDENSE matrix and perform

MatMatMult with A to get a resultant dense matrix B. I can now use
MatGetSubMatrices on B to access the non-local part required in my
subsequent calculations.

Post by Bikash Kanungo
2. Use MatMult to evaluate A*v_i for each of the vectors and provide

ghost padding through VecCreateGhost for the resultant vector.

Post by Bikash Kanungo
Which of the two is a more efficient way to perform the task? Also, if

there is a better way possible to achieve the same, then kindly let me know.

Post by Bikash Kanungo
Thanks,
Bikash
--
Bikash S. Kanungo
PhD Student
Computational Materials Physics Group
Mechanical Engineering
University of Michigan

[Jed Brown]

Post by Bikash Kanungo
Matrix A defined in my problem is of size NxN and I have N vectors v_i,
each of length N, which are to be multiplied by A. N is fairly small
10000-50000.

Your vectors form a square matrix, so I would use MatMatMult. If the
number of vectors was much smaller than their size (e.g., 10), then
alternatives might make sense.

[Bikash Kanungo]

Thanks for the input Jed.

Post by Jed Brown

Post by Bikash Kanungo
Matrix A defined in my problem is of size NxN and I have N vectors v_i,
each of length N, which are to be multiplied by A. N is fairly small
10000-50000.

Your vectors form a square matrix, so I would use MatMatMult. If the
number of vectors was much smaller than their size (e.g., 10), then
alternatives might make sense.