Delete with multiple indices is extremely slow--workaround?











up vote
11
down vote

favorite
1












Delete is unbelievably slow when deleting multiple elements from a non-packed array.



Is there a robust workaround that will work on any non-packed array?



inds = List /@ RandomSample[Range[100000], 50000];
Delete[Developer`FromPackedArray@Range[100000], inds]; // AbsoluteTiming
(* {17.8957, Null} *)


On packed arrays it performs as expected, but my array cannot be packed. It does not necessarily contain numbers.



inds = List /@ RandomSample[Range[100000], 50000];
Delete[Range[100000], inds]; // AbsoluteTiming
(* {0.005767, Null} *)




I did not try to test for this, but one possible explanation is that even when given multiple indices, Delete will delete elements one-by-one, re-allocating the array after each step. If someone feels like testing it, you can try to see if the timing is quadratic in the number of elements deleted.










share|improve this question




























    up vote
    11
    down vote

    favorite
    1












    Delete is unbelievably slow when deleting multiple elements from a non-packed array.



    Is there a robust workaround that will work on any non-packed array?



    inds = List /@ RandomSample[Range[100000], 50000];
    Delete[Developer`FromPackedArray@Range[100000], inds]; // AbsoluteTiming
    (* {17.8957, Null} *)


    On packed arrays it performs as expected, but my array cannot be packed. It does not necessarily contain numbers.



    inds = List /@ RandomSample[Range[100000], 50000];
    Delete[Range[100000], inds]; // AbsoluteTiming
    (* {0.005767, Null} *)




    I did not try to test for this, but one possible explanation is that even when given multiple indices, Delete will delete elements one-by-one, re-allocating the array after each step. If someone feels like testing it, you can try to see if the timing is quadratic in the number of elements deleted.










    share|improve this question


























      up vote
      11
      down vote

      favorite
      1









      up vote
      11
      down vote

      favorite
      1






      1





      Delete is unbelievably slow when deleting multiple elements from a non-packed array.



      Is there a robust workaround that will work on any non-packed array?



      inds = List /@ RandomSample[Range[100000], 50000];
      Delete[Developer`FromPackedArray@Range[100000], inds]; // AbsoluteTiming
      (* {17.8957, Null} *)


      On packed arrays it performs as expected, but my array cannot be packed. It does not necessarily contain numbers.



      inds = List /@ RandomSample[Range[100000], 50000];
      Delete[Range[100000], inds]; // AbsoluteTiming
      (* {0.005767, Null} *)




      I did not try to test for this, but one possible explanation is that even when given multiple indices, Delete will delete elements one-by-one, re-allocating the array after each step. If someone feels like testing it, you can try to see if the timing is quadratic in the number of elements deleted.










      share|improve this question















      Delete is unbelievably slow when deleting multiple elements from a non-packed array.



      Is there a robust workaround that will work on any non-packed array?



      inds = List /@ RandomSample[Range[100000], 50000];
      Delete[Developer`FromPackedArray@Range[100000], inds]; // AbsoluteTiming
      (* {17.8957, Null} *)


      On packed arrays it performs as expected, but my array cannot be packed. It does not necessarily contain numbers.



      inds = List /@ RandomSample[Range[100000], 50000];
      Delete[Range[100000], inds]; // AbsoluteTiming
      (* {0.005767, Null} *)




      I did not try to test for this, but one possible explanation is that even when given multiple indices, Delete will delete elements one-by-one, re-allocating the array after each step. If someone feels like testing it, you can try to see if the timing is quadratic in the number of elements deleted.







      list-manipulation performance-tuning






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited 18 hours ago









      xzczd

      25.7k468244




      25.7k468244










      asked yesterday









      Szabolcs

      158k13430923




      158k13430923






















          2 Answers
          2






          active

          oldest

          votes

















          up vote
          12
          down vote













          Exploiting the fact that Delete works fine on packed arrays, we can first construct an index vector, delete the unneeded indices, then finally use the remaining ones to index into the main array.



          arr = Developer`FromPackedArray@Range[100000];
          inds = List /@ RandomSample[Range[100000], 50000];

          Part[arr, Delete[Range@Length[arr], inds]]; // AbsoluteTiming
          (* {0.006371, Null} *)





          share|improve this answer

















          • 1




            Embarrassing for Delete that this method which ought to have terrible time complexity is so much faster... Have you tested some of the other things like delete on unpacked arrays?
            – b3m2a1
            yesterday








          • 3




            @b3m2a1 I don't think that this has larger complexity... Still it is pretty bad that Delete is not clever enough to do that automatically. I'd suggest to inform Wolfram Support.
            – Henrik Schumacher
            yesterday




















          up vote
          6
          down vote













          Why not use Part assignment (to Sequence) instead?



          arr = Developer`FromPackedArray@Range[100000];
          inds = List /@ RandomSample[Range[100000],50000];

          r1 = Part[arr, Delete[Range@Length[arr], inds]]; //RepeatedTiming
          (r2 = arr; r2[[Flatten @ inds]] = Sequence;) //RepeatedTiming

          r1 === r2



          {0.0059, Null}



          {0.0019, Null}



          True







          share|improve this answer





















          • Nothing is slightly faster than Sequence on my laptop.
            – Sjoerd C. de Vries
            yesterday












          • @SjoerdC.deVries and Carl, to fully reduce the array the procedure needs to be followed by r2;. Both Sequence and Nothing will now produce same timings as Part+Delete. p.s. to see what I mean try this: r = {1, 2}; r[[1]] = Nothing; Information@r
            – Kuba
            17 hours ago













          Your Answer





          StackExchange.ifUsing("editor", function () {
          return StackExchange.using("mathjaxEditing", function () {
          StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
          StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
          });
          });
          }, "mathjax-editing");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "387"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          convertImagesToLinks: false,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: null,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f187206%2fdelete-with-multiple-indices-is-extremely-slow-workaround%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          2 Answers
          2






          active

          oldest

          votes








          2 Answers
          2






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes








          up vote
          12
          down vote













          Exploiting the fact that Delete works fine on packed arrays, we can first construct an index vector, delete the unneeded indices, then finally use the remaining ones to index into the main array.



          arr = Developer`FromPackedArray@Range[100000];
          inds = List /@ RandomSample[Range[100000], 50000];

          Part[arr, Delete[Range@Length[arr], inds]]; // AbsoluteTiming
          (* {0.006371, Null} *)





          share|improve this answer

















          • 1




            Embarrassing for Delete that this method which ought to have terrible time complexity is so much faster... Have you tested some of the other things like delete on unpacked arrays?
            – b3m2a1
            yesterday








          • 3




            @b3m2a1 I don't think that this has larger complexity... Still it is pretty bad that Delete is not clever enough to do that automatically. I'd suggest to inform Wolfram Support.
            – Henrik Schumacher
            yesterday

















          up vote
          12
          down vote













          Exploiting the fact that Delete works fine on packed arrays, we can first construct an index vector, delete the unneeded indices, then finally use the remaining ones to index into the main array.



          arr = Developer`FromPackedArray@Range[100000];
          inds = List /@ RandomSample[Range[100000], 50000];

          Part[arr, Delete[Range@Length[arr], inds]]; // AbsoluteTiming
          (* {0.006371, Null} *)





          share|improve this answer

















          • 1




            Embarrassing for Delete that this method which ought to have terrible time complexity is so much faster... Have you tested some of the other things like delete on unpacked arrays?
            – b3m2a1
            yesterday








          • 3




            @b3m2a1 I don't think that this has larger complexity... Still it is pretty bad that Delete is not clever enough to do that automatically. I'd suggest to inform Wolfram Support.
            – Henrik Schumacher
            yesterday















          up vote
          12
          down vote










          up vote
          12
          down vote









          Exploiting the fact that Delete works fine on packed arrays, we can first construct an index vector, delete the unneeded indices, then finally use the remaining ones to index into the main array.



          arr = Developer`FromPackedArray@Range[100000];
          inds = List /@ RandomSample[Range[100000], 50000];

          Part[arr, Delete[Range@Length[arr], inds]]; // AbsoluteTiming
          (* {0.006371, Null} *)





          share|improve this answer












          Exploiting the fact that Delete works fine on packed arrays, we can first construct an index vector, delete the unneeded indices, then finally use the remaining ones to index into the main array.



          arr = Developer`FromPackedArray@Range[100000];
          inds = List /@ RandomSample[Range[100000], 50000];

          Part[arr, Delete[Range@Length[arr], inds]]; // AbsoluteTiming
          (* {0.006371, Null} *)






          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered yesterday









          Szabolcs

          158k13430923




          158k13430923








          • 1




            Embarrassing for Delete that this method which ought to have terrible time complexity is so much faster... Have you tested some of the other things like delete on unpacked arrays?
            – b3m2a1
            yesterday








          • 3




            @b3m2a1 I don't think that this has larger complexity... Still it is pretty bad that Delete is not clever enough to do that automatically. I'd suggest to inform Wolfram Support.
            – Henrik Schumacher
            yesterday
















          • 1




            Embarrassing for Delete that this method which ought to have terrible time complexity is so much faster... Have you tested some of the other things like delete on unpacked arrays?
            – b3m2a1
            yesterday








          • 3




            @b3m2a1 I don't think that this has larger complexity... Still it is pretty bad that Delete is not clever enough to do that automatically. I'd suggest to inform Wolfram Support.
            – Henrik Schumacher
            yesterday










          1




          1




          Embarrassing for Delete that this method which ought to have terrible time complexity is so much faster... Have you tested some of the other things like delete on unpacked arrays?
          – b3m2a1
          yesterday






          Embarrassing for Delete that this method which ought to have terrible time complexity is so much faster... Have you tested some of the other things like delete on unpacked arrays?
          – b3m2a1
          yesterday






          3




          3




          @b3m2a1 I don't think that this has larger complexity... Still it is pretty bad that Delete is not clever enough to do that automatically. I'd suggest to inform Wolfram Support.
          – Henrik Schumacher
          yesterday






          @b3m2a1 I don't think that this has larger complexity... Still it is pretty bad that Delete is not clever enough to do that automatically. I'd suggest to inform Wolfram Support.
          – Henrik Schumacher
          yesterday












          up vote
          6
          down vote













          Why not use Part assignment (to Sequence) instead?



          arr = Developer`FromPackedArray@Range[100000];
          inds = List /@ RandomSample[Range[100000],50000];

          r1 = Part[arr, Delete[Range@Length[arr], inds]]; //RepeatedTiming
          (r2 = arr; r2[[Flatten @ inds]] = Sequence;) //RepeatedTiming

          r1 === r2



          {0.0059, Null}



          {0.0019, Null}



          True







          share|improve this answer





















          • Nothing is slightly faster than Sequence on my laptop.
            – Sjoerd C. de Vries
            yesterday












          • @SjoerdC.deVries and Carl, to fully reduce the array the procedure needs to be followed by r2;. Both Sequence and Nothing will now produce same timings as Part+Delete. p.s. to see what I mean try this: r = {1, 2}; r[[1]] = Nothing; Information@r
            – Kuba
            17 hours ago

















          up vote
          6
          down vote













          Why not use Part assignment (to Sequence) instead?



          arr = Developer`FromPackedArray@Range[100000];
          inds = List /@ RandomSample[Range[100000],50000];

          r1 = Part[arr, Delete[Range@Length[arr], inds]]; //RepeatedTiming
          (r2 = arr; r2[[Flatten @ inds]] = Sequence;) //RepeatedTiming

          r1 === r2



          {0.0059, Null}



          {0.0019, Null}



          True







          share|improve this answer





















          • Nothing is slightly faster than Sequence on my laptop.
            – Sjoerd C. de Vries
            yesterday












          • @SjoerdC.deVries and Carl, to fully reduce the array the procedure needs to be followed by r2;. Both Sequence and Nothing will now produce same timings as Part+Delete. p.s. to see what I mean try this: r = {1, 2}; r[[1]] = Nothing; Information@r
            – Kuba
            17 hours ago















          up vote
          6
          down vote










          up vote
          6
          down vote









          Why not use Part assignment (to Sequence) instead?



          arr = Developer`FromPackedArray@Range[100000];
          inds = List /@ RandomSample[Range[100000],50000];

          r1 = Part[arr, Delete[Range@Length[arr], inds]]; //RepeatedTiming
          (r2 = arr; r2[[Flatten @ inds]] = Sequence;) //RepeatedTiming

          r1 === r2



          {0.0059, Null}



          {0.0019, Null}



          True







          share|improve this answer












          Why not use Part assignment (to Sequence) instead?



          arr = Developer`FromPackedArray@Range[100000];
          inds = List /@ RandomSample[Range[100000],50000];

          r1 = Part[arr, Delete[Range@Length[arr], inds]]; //RepeatedTiming
          (r2 = arr; r2[[Flatten @ inds]] = Sequence;) //RepeatedTiming

          r1 === r2



          {0.0059, Null}



          {0.0019, Null}



          True








          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered yesterday









          Carl Woll

          66k385171




          66k385171












          • Nothing is slightly faster than Sequence on my laptop.
            – Sjoerd C. de Vries
            yesterday












          • @SjoerdC.deVries and Carl, to fully reduce the array the procedure needs to be followed by r2;. Both Sequence and Nothing will now produce same timings as Part+Delete. p.s. to see what I mean try this: r = {1, 2}; r[[1]] = Nothing; Information@r
            – Kuba
            17 hours ago




















          • Nothing is slightly faster than Sequence on my laptop.
            – Sjoerd C. de Vries
            yesterday












          • @SjoerdC.deVries and Carl, to fully reduce the array the procedure needs to be followed by r2;. Both Sequence and Nothing will now produce same timings as Part+Delete. p.s. to see what I mean try this: r = {1, 2}; r[[1]] = Nothing; Information@r
            – Kuba
            17 hours ago


















          Nothing is slightly faster than Sequence on my laptop.
          – Sjoerd C. de Vries
          yesterday






          Nothing is slightly faster than Sequence on my laptop.
          – Sjoerd C. de Vries
          yesterday














          @SjoerdC.deVries and Carl, to fully reduce the array the procedure needs to be followed by r2;. Both Sequence and Nothing will now produce same timings as Part+Delete. p.s. to see what I mean try this: r = {1, 2}; r[[1]] = Nothing; Information@r
          – Kuba
          17 hours ago






          @SjoerdC.deVries and Carl, to fully reduce the array the procedure needs to be followed by r2;. Both Sequence and Nothing will now produce same timings as Part+Delete. p.s. to see what I mean try this: r = {1, 2}; r[[1]] = Nothing; Information@r
          – Kuba
          17 hours ago




















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Mathematica Stack Exchange!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          Use MathJax to format equations. MathJax reference.


          To learn more, see our tips on writing great answers.





          Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


          Please pay close attention to the following guidance:


          • Please be sure to answer the question. Provide details and share your research!

          But avoid



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f187206%2fdelete-with-multiple-indices-is-extremely-slow-workaround%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          Popular posts from this blog

          List directoties down one level, excluding some named directories and files

          list processes belonging to a network namespace

          list systemd RuntimeDirectory mounts