Delete with multiple indices is extremely slow--workaround?
up vote
11
down vote
favorite
Delete is unbelievably slow when deleting multiple elements from a non-packed array.
Is there a robust workaround that will work on any non-packed array?
inds = List /@ RandomSample[Range[100000], 50000];
Delete[Developer`FromPackedArray@Range[100000], inds]; // AbsoluteTiming
(* {17.8957, Null} *)
On packed arrays it performs as expected, but my array cannot be packed. It does not necessarily contain numbers.
inds = List /@ RandomSample[Range[100000], 50000];
Delete[Range[100000], inds]; // AbsoluteTiming
(* {0.005767, Null} *)
I did not try to test for this, but one possible explanation is that even when given multiple indices, Delete will delete elements one-by-one, re-allocating the array after each step. If someone feels like testing it, you can try to see if the timing is quadratic in the number of elements deleted.
list-manipulation performance-tuning
add a comment |
up vote
11
down vote
favorite
Delete is unbelievably slow when deleting multiple elements from a non-packed array.
Is there a robust workaround that will work on any non-packed array?
inds = List /@ RandomSample[Range[100000], 50000];
Delete[Developer`FromPackedArray@Range[100000], inds]; // AbsoluteTiming
(* {17.8957, Null} *)
On packed arrays it performs as expected, but my array cannot be packed. It does not necessarily contain numbers.
inds = List /@ RandomSample[Range[100000], 50000];
Delete[Range[100000], inds]; // AbsoluteTiming
(* {0.005767, Null} *)
I did not try to test for this, but one possible explanation is that even when given multiple indices, Delete will delete elements one-by-one, re-allocating the array after each step. If someone feels like testing it, you can try to see if the timing is quadratic in the number of elements deleted.
list-manipulation performance-tuning
add a comment |
up vote
11
down vote
favorite
up vote
11
down vote
favorite
Delete is unbelievably slow when deleting multiple elements from a non-packed array.
Is there a robust workaround that will work on any non-packed array?
inds = List /@ RandomSample[Range[100000], 50000];
Delete[Developer`FromPackedArray@Range[100000], inds]; // AbsoluteTiming
(* {17.8957, Null} *)
On packed arrays it performs as expected, but my array cannot be packed. It does not necessarily contain numbers.
inds = List /@ RandomSample[Range[100000], 50000];
Delete[Range[100000], inds]; // AbsoluteTiming
(* {0.005767, Null} *)
I did not try to test for this, but one possible explanation is that even when given multiple indices, Delete will delete elements one-by-one, re-allocating the array after each step. If someone feels like testing it, you can try to see if the timing is quadratic in the number of elements deleted.
list-manipulation performance-tuning
Delete is unbelievably slow when deleting multiple elements from a non-packed array.
Is there a robust workaround that will work on any non-packed array?
inds = List /@ RandomSample[Range[100000], 50000];
Delete[Developer`FromPackedArray@Range[100000], inds]; // AbsoluteTiming
(* {17.8957, Null} *)
On packed arrays it performs as expected, but my array cannot be packed. It does not necessarily contain numbers.
inds = List /@ RandomSample[Range[100000], 50000];
Delete[Range[100000], inds]; // AbsoluteTiming
(* {0.005767, Null} *)
I did not try to test for this, but one possible explanation is that even when given multiple indices, Delete will delete elements one-by-one, re-allocating the array after each step. If someone feels like testing it, you can try to see if the timing is quadratic in the number of elements deleted.
list-manipulation performance-tuning
list-manipulation performance-tuning
edited 18 hours ago
xzczd
25.7k468244
25.7k468244
asked yesterday
Szabolcs
158k13430923
158k13430923
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
up vote
12
down vote
Exploiting the fact that Delete works fine on packed arrays, we can first construct an index vector, delete the unneeded indices, then finally use the remaining ones to index into the main array.
arr = Developer`FromPackedArray@Range[100000];
inds = List /@ RandomSample[Range[100000], 50000];
Part[arr, Delete[Range@Length[arr], inds]]; // AbsoluteTiming
(* {0.006371, Null} *)
1
Embarrassing forDeletethat this method which ought to have terrible time complexity is so much faster... Have you tested some of the other things like delete on unpacked arrays?
– b3m2a1
yesterday
3
@b3m2a1 I don't think that this has larger complexity... Still it is pretty bad thatDeleteis not clever enough to do that automatically. I'd suggest to inform Wolfram Support.
– Henrik Schumacher
yesterday
add a comment |
up vote
6
down vote
Why not use Part assignment (to Sequence) instead?
arr = Developer`FromPackedArray@Range[100000];
inds = List /@ RandomSample[Range[100000],50000];
r1 = Part[arr, Delete[Range@Length[arr], inds]]; //RepeatedTiming
(r2 = arr; r2[[Flatten @ inds]] = Sequence;) //RepeatedTiming
r1 === r2
{0.0059, Null}
{0.0019, Null}
True
Nothingis slightly faster thanSequenceon my laptop.
– Sjoerd C. de Vries
yesterday
@SjoerdC.deVries and Carl, to fully reduce the array the procedure needs to be followed byr2;. Both Sequence and Nothing will now produce same timings as Part+Delete. p.s. to see what I mean try this:r = {1, 2}; r[[1]] = Nothing; Information@r
– Kuba♦
17 hours ago
add a comment |
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
12
down vote
Exploiting the fact that Delete works fine on packed arrays, we can first construct an index vector, delete the unneeded indices, then finally use the remaining ones to index into the main array.
arr = Developer`FromPackedArray@Range[100000];
inds = List /@ RandomSample[Range[100000], 50000];
Part[arr, Delete[Range@Length[arr], inds]]; // AbsoluteTiming
(* {0.006371, Null} *)
1
Embarrassing forDeletethat this method which ought to have terrible time complexity is so much faster... Have you tested some of the other things like delete on unpacked arrays?
– b3m2a1
yesterday
3
@b3m2a1 I don't think that this has larger complexity... Still it is pretty bad thatDeleteis not clever enough to do that automatically. I'd suggest to inform Wolfram Support.
– Henrik Schumacher
yesterday
add a comment |
up vote
12
down vote
Exploiting the fact that Delete works fine on packed arrays, we can first construct an index vector, delete the unneeded indices, then finally use the remaining ones to index into the main array.
arr = Developer`FromPackedArray@Range[100000];
inds = List /@ RandomSample[Range[100000], 50000];
Part[arr, Delete[Range@Length[arr], inds]]; // AbsoluteTiming
(* {0.006371, Null} *)
1
Embarrassing forDeletethat this method which ought to have terrible time complexity is so much faster... Have you tested some of the other things like delete on unpacked arrays?
– b3m2a1
yesterday
3
@b3m2a1 I don't think that this has larger complexity... Still it is pretty bad thatDeleteis not clever enough to do that automatically. I'd suggest to inform Wolfram Support.
– Henrik Schumacher
yesterday
add a comment |
up vote
12
down vote
up vote
12
down vote
Exploiting the fact that Delete works fine on packed arrays, we can first construct an index vector, delete the unneeded indices, then finally use the remaining ones to index into the main array.
arr = Developer`FromPackedArray@Range[100000];
inds = List /@ RandomSample[Range[100000], 50000];
Part[arr, Delete[Range@Length[arr], inds]]; // AbsoluteTiming
(* {0.006371, Null} *)
Exploiting the fact that Delete works fine on packed arrays, we can first construct an index vector, delete the unneeded indices, then finally use the remaining ones to index into the main array.
arr = Developer`FromPackedArray@Range[100000];
inds = List /@ RandomSample[Range[100000], 50000];
Part[arr, Delete[Range@Length[arr], inds]]; // AbsoluteTiming
(* {0.006371, Null} *)
answered yesterday
Szabolcs
158k13430923
158k13430923
1
Embarrassing forDeletethat this method which ought to have terrible time complexity is so much faster... Have you tested some of the other things like delete on unpacked arrays?
– b3m2a1
yesterday
3
@b3m2a1 I don't think that this has larger complexity... Still it is pretty bad thatDeleteis not clever enough to do that automatically. I'd suggest to inform Wolfram Support.
– Henrik Schumacher
yesterday
add a comment |
1
Embarrassing forDeletethat this method which ought to have terrible time complexity is so much faster... Have you tested some of the other things like delete on unpacked arrays?
– b3m2a1
yesterday
3
@b3m2a1 I don't think that this has larger complexity... Still it is pretty bad thatDeleteis not clever enough to do that automatically. I'd suggest to inform Wolfram Support.
– Henrik Schumacher
yesterday
1
1
Embarrassing for
Delete that this method which ought to have terrible time complexity is so much faster... Have you tested some of the other things like delete on unpacked arrays?– b3m2a1
yesterday
Embarrassing for
Delete that this method which ought to have terrible time complexity is so much faster... Have you tested some of the other things like delete on unpacked arrays?– b3m2a1
yesterday
3
3
@b3m2a1 I don't think that this has larger complexity... Still it is pretty bad that
Delete is not clever enough to do that automatically. I'd suggest to inform Wolfram Support.– Henrik Schumacher
yesterday
@b3m2a1 I don't think that this has larger complexity... Still it is pretty bad that
Delete is not clever enough to do that automatically. I'd suggest to inform Wolfram Support.– Henrik Schumacher
yesterday
add a comment |
up vote
6
down vote
Why not use Part assignment (to Sequence) instead?
arr = Developer`FromPackedArray@Range[100000];
inds = List /@ RandomSample[Range[100000],50000];
r1 = Part[arr, Delete[Range@Length[arr], inds]]; //RepeatedTiming
(r2 = arr; r2[[Flatten @ inds]] = Sequence;) //RepeatedTiming
r1 === r2
{0.0059, Null}
{0.0019, Null}
True
Nothingis slightly faster thanSequenceon my laptop.
– Sjoerd C. de Vries
yesterday
@SjoerdC.deVries and Carl, to fully reduce the array the procedure needs to be followed byr2;. Both Sequence and Nothing will now produce same timings as Part+Delete. p.s. to see what I mean try this:r = {1, 2}; r[[1]] = Nothing; Information@r
– Kuba♦
17 hours ago
add a comment |
up vote
6
down vote
Why not use Part assignment (to Sequence) instead?
arr = Developer`FromPackedArray@Range[100000];
inds = List /@ RandomSample[Range[100000],50000];
r1 = Part[arr, Delete[Range@Length[arr], inds]]; //RepeatedTiming
(r2 = arr; r2[[Flatten @ inds]] = Sequence;) //RepeatedTiming
r1 === r2
{0.0059, Null}
{0.0019, Null}
True
Nothingis slightly faster thanSequenceon my laptop.
– Sjoerd C. de Vries
yesterday
@SjoerdC.deVries and Carl, to fully reduce the array the procedure needs to be followed byr2;. Both Sequence and Nothing will now produce same timings as Part+Delete. p.s. to see what I mean try this:r = {1, 2}; r[[1]] = Nothing; Information@r
– Kuba♦
17 hours ago
add a comment |
up vote
6
down vote
up vote
6
down vote
Why not use Part assignment (to Sequence) instead?
arr = Developer`FromPackedArray@Range[100000];
inds = List /@ RandomSample[Range[100000],50000];
r1 = Part[arr, Delete[Range@Length[arr], inds]]; //RepeatedTiming
(r2 = arr; r2[[Flatten @ inds]] = Sequence;) //RepeatedTiming
r1 === r2
{0.0059, Null}
{0.0019, Null}
True
Why not use Part assignment (to Sequence) instead?
arr = Developer`FromPackedArray@Range[100000];
inds = List /@ RandomSample[Range[100000],50000];
r1 = Part[arr, Delete[Range@Length[arr], inds]]; //RepeatedTiming
(r2 = arr; r2[[Flatten @ inds]] = Sequence;) //RepeatedTiming
r1 === r2
{0.0059, Null}
{0.0019, Null}
True
answered yesterday
Carl Woll
66k385171
66k385171
Nothingis slightly faster thanSequenceon my laptop.
– Sjoerd C. de Vries
yesterday
@SjoerdC.deVries and Carl, to fully reduce the array the procedure needs to be followed byr2;. Both Sequence and Nothing will now produce same timings as Part+Delete. p.s. to see what I mean try this:r = {1, 2}; r[[1]] = Nothing; Information@r
– Kuba♦
17 hours ago
add a comment |
Nothingis slightly faster thanSequenceon my laptop.
– Sjoerd C. de Vries
yesterday
@SjoerdC.deVries and Carl, to fully reduce the array the procedure needs to be followed byr2;. Both Sequence and Nothing will now produce same timings as Part+Delete. p.s. to see what I mean try this:r = {1, 2}; r[[1]] = Nothing; Information@r
– Kuba♦
17 hours ago
Nothing is slightly faster than Sequence on my laptop.– Sjoerd C. de Vries
yesterday
Nothing is slightly faster than Sequence on my laptop.– Sjoerd C. de Vries
yesterday
@SjoerdC.deVries and Carl, to fully reduce the array the procedure needs to be followed by
r2;. Both Sequence and Nothing will now produce same timings as Part+Delete. p.s. to see what I mean try this: r = {1, 2}; r[[1]] = Nothing; Information@r– Kuba♦
17 hours ago
@SjoerdC.deVries and Carl, to fully reduce the array the procedure needs to be followed by
r2;. Both Sequence and Nothing will now produce same timings as Part+Delete. p.s. to see what I mean try this: r = {1, 2}; r[[1]] = Nothing; Information@r– Kuba♦
17 hours ago
add a comment |
Thanks for contributing an answer to Mathematica Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f187206%2fdelete-with-multiple-indices-is-extremely-slow-workaround%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown